RS Web API bugs

[majstang]

Finally had some time to dig deeper into the WebAPI issues I discovered after version 1.24 and 1.25 where several API changes were done. Hopefully also give the new developers a first test in RS WebAPI programming.

 

These are the API changes done in version 1.24:

 

 

WebAPI

• Add: "getchannelsxml.html" API call added.
  This new call serves a channel list based on several parameters in a UTF8 coded XML format. The standard convention for RS API calls applies.
  The parameters:
  • No parameter
    A minimalistic channel list is returned with the appropriate root and category nodes.
    A minimalistic channel entry consists of channel number, name, EPGID and favoriteID (=channelID) (without the name part).
  • rootsonly=1
    Only root nodes are returned.
  • groupsonly=1
    Root nodes and according category nodes are returned.
  • root=[Root name]
    Only the entries matching the root name are returned.
  • group=[Group Name]
    Only the matching entries for the category are returned.
  • number=[Channel number]
    Only the data for the channel with the channel number is returned.
  • logo=1
    The channel nodes contain the URL for the channel logo (if found). Attention: This may take a lot of time on the first call for extensive channel lists.
  • tuner=1
    The channel nodes contain the tuner record.
  • rtsp=1
    The RTSP(base)URL is returned as a node below the root node (if RTSP is activated) and the channel nodes contain the RTSP parameter which have to be combined with the rtspURL for a RTSP SETUP call.
  • upnp=1
    The UPnPURL for LiveTV streaming is returned as a node below the root node (if UPnP is activated). The real URL for a channel can be calculated as upnpURL + [Channel number] + ".ts"
  • subchannels=1
    If existing the audio sub channel are returned as „subchannel" nodes below the „channel" node. A subchannel entry consist of the channel name and the channel(favorite)ID. The other (for external programs interesting) parameters are covered by the parent channel node.
• Change: "recordings.html" API reworked and optimized.
  Attention: From this version on this function only works with UTF-8 encoded data.
  Also some unused entries have been removed. Some entries only appear if their value is >0 (minimumage, content).
  A new node has been added below the root node: "serverURL" which contains the BaseURL for the recordings streaming server. To calculate the actual streaming URL of a file you can use: serverURL + id + ".ts".
  Several GET URL parameters have been added:
  • nofilename=1
    No (network) file name nodes are returned. → smaller download.
    If nofilename is used a new property is added to the recording node: format="[file extension]". This is simply the file extension of the recording (.ts, .mpg etc.).
  • nodesc=1
    The "desc" node is suppressed. → smaller download.
  • images=1
    The baseURL for the thumbnails of the recordings is added to the root node as "imageURL".
    The "recording" node contains the "image" subnode with the file name of the thumbnail.
    If thumbnails are disabled or no thumbnail is available, this node will be omitted.
    To download the image just combine the node value with the imageURL. You can use the "height" parameter (see above → GET channel logos) to downscale the image.
  • id=[recid]
    Only the entry of the recording with "ID" is returned (if it exists). Can be combined with the above parameters.
• Change: "timeradd.html" and "timeredit.html" API.
  Several GET URL parameters have been added:
  • eit=1
    Enables the recording of the Actual EIT EPG Current/Next.
  • monitorpdc=1
    Enables the PDC monitoring for EPG changes. A valid PDC must be exist/supplied.
  • pdc=[PDC]
    A valid PDC value.
  • monforrec=[0,1, 2]
    The monitor for recording value. 0 – None, 1 – VBI PDC, 2 – EIT Running Status. A valid PDC value must exists.

 

...and these for version 1.25:

 

 

• Fix: WebAPI: "getchannelsxml.html" Problem with the tuner entry fixed.

• Add: WebAPI: "getchannelsxml.html" flags property added to the base channel entry.

• Add: WebAPI/„status.html“: Added the <epglang> entry. It shows the EPG language selected by the user.

• Add: WebAPI/„status.html“: Added the <recfiles> entry. It reflects the number of existing recording files.

• Add: WebAPI/„status.html“: URL parameter „norecdirs=1“ added. If set the RS does not list the recording folders.

• Add: WebAPI/„getchannelsxml.html“: URL query parameter „id=[ChannelID 32/64bit]“ added.

• Add: WebAPI/„getchannelsxml.html“: URL query parameter „epgid=[ChannelEPGID]“ added.

• Add: WebAPI: New documentation http://en.DVBViewer....Service_Web_API (work in progress).

 

Now the first issue appeared in RS 1.24 and more precise regards after recording tasks (process timer) no longer are counted in the recordcount variable. When a recording finishes and the after recording task fires, recordcount is blank. Not even a zero in there. It last worked in RS 1.23.1, which I just confirmed. Looking at the changelog for 1.24, Lars never made any alterations for the "status.html" API. The changes made was for "recordings.html" and "getchannelsxml.html" only.

 

Now at the time I was quite happy about that change cuz I have been advocating many times having separate countvariables for recordingtimers and process/internal task timers. Two reasons for it:

1. Having process/internal task timers counting up recordcount and interfer with the actual status for recordings makes it very hard to write reliable scripts that is supposed to monitor the ongoing recording status and execute various scripttasks depending on recordcount status. Maybe Lars had started doing the separation of timercount variables, but never got to finish it. Sadly as it is now it's just broken and unfortunately we never got the chance to find out what intentions Lars had with the change (maybe it was just a small mistake and no ulterior intentions behind it). In any case something must be done fixing it.

2. Scheduled process/internal task timers still counts up the recordcount value (which strengthen the mistake theory) and that makes the RS trayicon flash red as if a recording just started. I was questioning this behaviour in the past and do still consider it as unnecessary. Only recordings should turn the tray icon red. The colour changing of the tray icon is eniterly linked to the recordcount variable. My suggestion was if linking this function to the new statusvariable for process/internal task timers, which possible could turn the icon green for exsample when these kind of timers fires.

 

I suspect these changes would have implicated a bit of re-writes of RS, do think it was not viable considering the workload I think Lars had at that time. Now it makes me sad to see the state the API is in now. Please, try to find some time to go over it and there is a bunch of suggestions waiting to be realized in the Scripting Lounge Recording Service API thread. The API has to be completed sometime, so why not start hacking away.

 

Second issue in WebAPI status.html is the clientcount variable just behaves peculiar. Also introduced in RS 1.24.

This issue have been discussed here:

http://www.DVBViewer.tv/forum/topic/52798-webapi-clientcount-bug/?p=390361

 

Regards

majstang

 

[majstang]

After some more testing with RS 1.29 I have to make some clarifications. For all recordings scheduled with autotimer or manual scheduling via webinterface (without any manipulation of start and stoptime) the after recording task counts up recordcount by 1 as it should, but it also counts up clientcount by the same value. Recordings manually started in the middle of show (mening show has already aired for a while when I start recording) for these ones after recording tasks blanks out both recordcount and clientcount values. Same if starting a recording in the middle and editing/advancing stoptime (easier for testpurposes). So the bug seems only to show itself under specific circumstances.

 

Strange it blanks out clientcount as well. It's as if clientcount and recordcount uses the same API instructions/code. Maybe that is the problem here?

 

EDIT:

Found an another case where blanking of both recordccount and clientcount values occurs. If having two simultaneously going recordings with same stoptime (scheduled with either autotimer or manual scheduling via webinterface without any manipulation of start and stoptime) the after recording task blanks out both values.

Edited August 2, 2014 by majstang

[Griga]

Thanks for the analysis. I'll add it to my bookmarks. It may become very useful in future.

 

However

 

so why not start hacking away.

 

will not be possible in near future. The Recording service consists of 420.000 lines of undocumented code. Every change is very time-consuming, because it requires hours of code-reading, analysing and searching for references to other parts in order to avoid unwanted side effects. It's like travelling in an unknown country without a map. One has to move extremely carefully. And up to now I haven't investigated the API at all.

 

Of course, with growing experience and understanding for the internal structures more will become possible. But it will take time...

[majstang]

Totally understandable Griga! It's a huge undertaking unexpectedly falling in your lap. I have no doubt if anyone is going to succeed handling it, it's you. Takes a well-structured man for it and that seems to be in your nature, if judging from your actions on this forum. However unwanted side effects are impossible to avoid, even large ones can be mended. Hope that won't keep you from experimenting, even wildly sometimes. RS is after all just a beta and if the bugs are too big and severely effects the functionality, there is always the option to go back to previous version if a quick fix cant be found.

 

Best regards

majstang

[neophyte]

It's a huge undertaking unexpectedly falling in your lap. I have no doubt if anyone is going to succeed handling it, it's you. Takes a well-structured man for it and that seems to be in your nature, if judging from your actions on this forum.

+1

[Griga]

When a recording finishes and the after recording task fires, recordcount is blank.

 

Having some insight now, I tried to reproduce the issue, but without success. As "blank" value I expected something like

 

<status>

<recordcount></recordcount>

 

or

 

<status>

<recordcount/>

 

and did the following with RS 1.30:

 

(1) Selected a currently running radio programme on the Channel EPG page, clicked the red button -> timer window opens.

(2) Selected an after recording task (conversion to mp3 with ffmpeg), clicked save -> recording starts immediately

(3) Edited the timer, advanced the stop time by 2 minutes, clicked save.

(4) Requested the api/status.html in Firefox. Got the expected xml output (recordcount and clientcount = 1)

(5) Requested it again after the recording and the task were done (got no opportunity to perform the request while the task was executed, because it happened too fast) and got the following output in Firefox:

 

<status>

<recordcount>0</recordcount>

<clientcount>0</clientcount>

etc

 

Some variations of the steps above (according to your report) didn't change anything.

 

In the RS code the number is inserted by calling the Delphi runtime IntToStr function:

 

IntToStr(RecordCount)

 

which is a simple Integer -> String conversion. RecordCount is an unsigned 32 bit integer. AFAIK it is impossible that this function returns an empty string. I even tried to set RecordCount to odd values like 0xFFFFFFFF, which yielded 4294967295 in the xml output, but no blank value.

 

So I guess the value does not get lost in the RS, but later when transmitting / receiving / reading / evaluating the xml. How did you check the API output? Did you countercheck it with something else?

 

Please provide clear step by step instructions for reproducing issues in your next report(s) (I hope it won't be many), like my enumeration above. In your report there is important information missing, e.g. the exact meaning of "blank" (example needed), how you scheduled the recording, how and at which point of time you requested, received and checked the API output.

[majstang]

Hi Griga!

 

Nice to see you digging into this

I use autohotkey IniWrite to save the recordcount and clientcount variable contents. I added a second save method just to rule out one of them is reason for failure. The after recording task fires up the script containing this code. I quickly threw together a short version with the code needed to reproduce the "blanking" issue, pasted in bottom of post. Perhaps you would be interested in installing autohotkey yourself to test it? If so download and install from here: http://ahkscript.org/download/ahk-install.exe

 

Note, tests were done with RS 1.29. Since no API changes for this field has been done on 1.30 I believe the results would be the same if using that RS version.

 

Steps to reproduce:

(1) Install autohotkey

(2) Copy script into a texteditor, preferrably Notepad++

(3) Fill in your user specifics, such as username, password, port and change paths for IniWrite

(4) Save script and compile it (rightclick script in the explorer and choose compile). Doubleclick to start script.exe and verify it's working (creating the .ini and RS_status.xml and also check if it has contents)

(5) Create a new RS task and assign the script.exe

(6) Now from Timeline schedule a recording but change stoptime so it stops recording a couple of minutes from nowtime and apply the new task to it.

(7) Repeat step 6, but for an another show and choose the exact same stoptime and task. Two recordings with exact same stoptime has to be running to reproduce the issue.

 

Now when both recordings stops they fire their respective after recording task approximately 2 seconds appart.

 

(8) In my case when checking out the RS_Status.xml it is totally empty and Keys for recordcount and clientcount in the ini is also empty (blank).

 

Note, when initializing recordings like this from Timeline and advancing stoptime (with same stoptime), these recordings never gets the red frame around its EPG entry. Also in this particular case RS takes unusually long time to fire the after recording task and revert to idle state compared to if doing a single recording where the blanking never occurs.

 

Hmm...the code tag button seems to be removed so I'll use a quote instead:

 

 

FormatTime, TimeVar,, HH:mm:ss ;Current local time.

FormatTime, DateVar,, WDay ; 1=Sunday, 2=Monday etc.

Recording_Service_API_URL := "http://username:password@127.0.0.1:8075" ; You may change the port number also cuz I do not use the default one.

recordingstatus := URLToVar(Recording_Service_API_URL . "/api/status.html")

;;--MSXML DOM parsing method---------------

RS_Status := loadXML(recordingstatus)

RS_StatusVar := RS_Status.selectSingleNode("/status/recordcount").text

RS_ClientCountStatus := RS_Status.selectSingleNode("/status/clientcount").text

;MsgBox recordcount= %RS_StatusVar%`n clientcount= %RS_ClientCountStatus% ; msgbox cant be activated if fire script from an after recording task

IniWrite, %DateVar%, D:\TV\test\RecService Status.ini, Day_of_Week, Key

IniWrite, %TimeVar%, D:\TV\test\RecService Status.ini, CurrentTime, Key

IniWrite, %RS_StatusVar%, D:\TV\test\RecService Status.ini, RS_Status, Key

IniWrite, %RS_ClientCountStatus%, D:\TV\test\RecService Status.ini, RS_ClientCountStatus, Key

RS_Status.save("RS_Status.xml")

ExitApp

URLToVar(URL) {

ComObjError(0)

WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")

WebRequest.Open("GET", URL)

WebRequest.Send()

Return WebRequest.ResponseText()

, ComObjError(0)

}

;------------------------------------------------------------------------------------------------------------------

loadXML(ByRef data) {

o := ComObjCreate("MSXML2.DOMDocument.6.0")

o.async := false

o.preserveWhiteSpace := true

o.loadXML(data)

return o

}

 

Regards

majstang

Edited October 17, 2014 by majstang

[nuts]

Can't reproduce this too.

Could you please post your status.xml?

[Griga]

Steps to reproduce:

(1) Install autohotkey...

 

Takes too much time to investigate. Can you reproduce the issue by simply using a browser?

 

Could you please post your status.xml?

 

+1. It's still unclear how the output really looks like.

[majstang]

 

Takes too much time to investigate. Can you reproduce the issue by simply using a browser?

 

Ok, then we'll have to leave this bug as it is for now. It doesn't affect any actual recording functionality, so if short on time to investigate, it can wait.

Well, I can't see any way to reproduce it in a browser! The URLToVar function takes a snapshot of the status.xml (can be found in the RS_Status.xml the script produces) in the exact moment after the after recording task fires and stores it in a variable. How to accomplish that kind of timing with just a browser, I have no idea. Btw, I just tested with RS 1.30 and the issue persists. Last worked in RS 1.23.1.

 

 

Can't reproduce this too.

Could you please post your status.xml?

 

@nuts, did you do all the steps? Maybe I forgot some vital information in the steps? ((i really don't have time for this right now but can't stay away )) After some more testing (quick tests) this is the result:

 

If having one recording going or two with different stoptime the URLToVar function downloads this when the after recording task fires:

 

 

<?xml version="1.0" encoding="utf-8"?><status><recordcount>1</recordcount><clientcount>1</clientcount><epgudate>0</epgudate><epgbefore>1</epgbefore><epgafter>2</epgafter><recfiles>1246</recfiles><timezone>120</timezone><defafterrecord>0</defafterrecord><epglang>swe</epglang><recfolders><folder size="2000325439488" free="282976489472">D:\TV\Recording Service</folder></recfolders></status>

 

Hmm...im getting mixed results trying to reproduce my earlier results for this next case and it looks like i have to revise my earlier statements for now. I really have to go over my results again when having more time. I got one case were the issue showed up and an another where i got ok results (in quote below), so it seems some other parameter which im not aware of plays a role. Anyway case is:

If having two recordings going with same stoptime, with NO manual manipulation of stoptime, it stops according to what the EPG stipulates.

The URLToVar function produces this:

 

 

<?xml version="1.0" encoding="utf-8"?><status><recordcount>2</recordcount><clientcount>2</clientcount><epgudate>0</epgudate><epgbefore>1</epgbefore><epgafter>2</epgafter><recfiles>1248</recfiles><timezone>120</timezone><defafterrecord>0</defafterrecord><epglang>swe</epglang><recfolders><folder size="2000325439488" free="282167058432">D:\TV\Recording Service</folder></recfolders></status>

 

So the only cases where issue appears, which is reproducable with certainty, is if having two or more recordings going with same stoptime, made by manual manipulation of stoptime (I advance the stoptime to end earlier than EPG states just for quicker testing of my scripts). In such a case if the two after recording tasks fires simultaneoulsy (well RS triggers the first task and fires the second task 2 seconds later) status.xml is empty. No xml formatting, no xml tags what so ever. It's just empty.

 

 

 

Edited October 17, 2014 by majstang

[Griga]

 

Ok, that's a different thing. Not certain values are empty, but the whole XML if retrieved at a certain point of time, which means, simply nothing is sent. Is this correct?

 

Sometimes nothing tells more than many words. Very philosophical...

[nuts]

So the only cases where issue appears, which is reproducable with certainty, is if having two or more recordings going with same stoptime, made by manual manipulation of stoptime (I advance the stoptime to end earlier than EPG states just for quicker testing of my scripts). In such a case if the two after recording tasks fires simultaneoulsy (well RS triggers the first task and fires the second task 2 seconds later) status.xml is empty. No xml formatting, no xml tags what so ever. It's just empty.

 

I am pretty sure there is no status.xml return at all! Your problem is here:

 

URLToVar(URL) {

ComObjError(0)

WebRequest := ComObjCreate("WinHttp.WinHttpRequest.5.1")

WebRequest.Open("GET", URL)

WebRequest.Send()

Return WebRequest.ResponseText()

, ComObjError(0)

}

 

 

Some Errorhandling could show more.

Edited October 17, 2014 by nuts

[majstang]

Ok, that's a different thing. Not certain values are empty, but the whole XML if retrieved at a certain point of time, which means, simply nothing is sent. Is this correct?

 

Sometimes nothing tells more than many words. Very philosophical...

 

Yes, that is correct! The RS API is somehow blocked for the whole duration of the after recording task run time and cant recieve any calls. It is confusing why the blanking always occurs on recording timers where i have messed with stoptime. For autotimer produced recording timers where start- and stoptime are stipulated by the EPG, this is not a problem. Not one API access failure so far there.

 

Alright, I definitely have to do some more testing to see if I can find out more.

 

 

 

I am pretty sure there is no status.xml return at all! Your problem is here:

 

 

Some Errorhandling could show more.

 

I can definitely not rule out anything at this point. Maybe you are right, but my gut feeling says RS, cuz if you were right i would see the blanking in every case where two or more timers finishes simultaneously and im not. I will see if i can find out more with some improved logging.

[majstang]

Went back to RS 1.23.1 and did some blanking issue tests and it goes on there as well. So after extensive testing it seems nuts are right this particular issue must be from a bug in my URLToVar function. Did try my old function i was using way back and it works better. No blanking but there is other issues with this one.

Hmm...hunting bugs is so confusing sometimes

 

The only bug in RS is clientcount is broken and gets the exact value as recordcount has, no matter if a client is running or not.

[Griga]

The blanking issue seems to be a timing problem. Launching two after recording tasks at the same time may have a siginificant impact on the response time. Maybe some timeout is too short... dunno. I would try something like (pseudo code):

 

i := 0

while not GetXML and (i < 3) do

i := i + 1

Sleep(100) //ms

whileend

[majstang]

Yes, it is a timing problem. Two simultaneously launched after recording task launches two separate instances of the URLToVar function, which both request the status.xml in a very short timeframe. RS seems to queue the after recording tasks, so not all triggers at the exact same moment. My tests shows the tasks triggers every 2 seconds under normal circumstances. In this case that goes wrong, somehow the webserver response from first URLToVar call stalls everything. The log shows task 1 triggers 1 second after stoptime and task 2 triggers 30 seconds after that. It takes approximately 65 seconds after stoptime until RS regains idle state. That time frame is definitely not right.

 

Im not sure if implementing a timeout like your above sample would help in this case. That is because of the fact that the after recording tasks launches two separate intances of the URLToVar script. If both tasks had been launched one single instance a timeout would have been effective. I can of cuz block that two instances of this same script would be allowed to run, but if doing that the script will never recieve the second "{SOURCE_FILE}" parameter and that recording wont be passed on to commercial removal. Huh, this is immensely tricky

Edited October 18, 2014 by majstang

[majstang]

Oh, I have to "strike while the iron is hot" when you are working on the API at the moment, to ask if 2. in first post in this topic is feasible or totally impossible to realize? Scripting some kind of reliable recording status monitoring function based on the API status.xml is impossible, especially when letting after recording tasks (processtimers) count up the recordcount value.

 

2. After recording tasks and other scheduled process/internal task timers still counts up the recordcount value and that makes the RS trayicon flash red as if a recording just started. I was questioning this behaviour in the past and do still consider it as unnecessary. Only recordings should turn the tray icon red. The colour changing of the tray icon is entirely linked to the recordcount variable. My suggestion was if linking this function to the new statusvariable for process/internal task timers, which possible could turn the icon green for example when these kind of timers fires.

Edited October 19, 2014 by majstang

[Griga]

The counters will probably be looked after next time when the Recording Service gets the focus. nuts has already done some useful researches in this respect by using a self-made debug tool, and we will come back to it.

 

However, after the RS bugfix release is up and (hopefully) stable, first there are other things to do resp. releases to be prepared.

[majstang]

Very promising news Both thumbs up for Griga and nuts for a mighty fine job you are doing