long standing EPG bug - Hong Kong TVB

[leeyc0]

This bug annoyed me from the first day I started to use the DVBViewer Recording Service. Below is the detailed description.

 

In my city (Hong Kong) there are two TV stations, named TVB and ATV. This problem only happens for EPG broadcasted by TVB.

 

The EPG items keep gradually disappearing if I keep the service running alone. The only way to update the EPG is to restart DVBViewer Recording Service, and manually trigger EPG scan in Tasks. However, even I re-fetched the EPG, it still keeps disappearing as time go on.

 

How can I provide more debug information for you?

[leeyc0]

Attached support.zip

support.zip

[Griga]

Most likely due to non DVB compliant EPG data (re-using event IDs that are supposed to be unique within a service). It's the broadcasters's fault and affects all software that stores EPG data for a longer period in order to make it accessible when you are not receiving the channel resp. frequency. We already experienced it with European broadcasters. After calling their attention to it they promised to fix it...

[Mr. Man-wai Chang]

I like to add my vote to this bug. Here is a screenshot of the error in Recording Service.

It seems that the bug happened right after a recording of TVB channel was started. Not too sure.

[leeyc0]

@griga: It seems to be the case. I just had a dump on broadcast stream dump using TransEdit, and I discovered that TVB EPG event ID is around 1-200, which is absolutely abnormal. (On the other hand, ATV EPG event ID is on the order of ten thousands.) I will have broadcast stream dump for a few more days to confirm this.

 

@Mr. Man-wai Chang: As griga said, it is not a bug in DVBViewer, but rather a misbahaviour from TVB.

 

@griga: Although this is the problem of the TV station (TVB), which they are supposed to fix, but in reality it is impossible to have their attention about this problem. Is it possible to have a workaround for this? (The TV station is notorious of being crappy in many areas, albeit being the largest TV station in Hong Kong -- I just heard from a friend inside the TV station that they are still using VHS tapes in some departments!!!!!)

Edited June 10, 2012 by leeyc0

[Mr. Man-wai Chang]

@Mr. Man-wai Chang: As griga said, it is not a bug in DVBViewer, but rather a misbahaviour from TVB.

@griga: Although this is the problem of the TV station (TVB), .... Is it possible to have a workaround for this?

If you emailed TVB's tech support, I believe the engineers there would just blame DVBViewer Pro. They would say TVB's EPG is working fine in hardware set-top boxes. So it's only the problem of DVBViewer, or maybe those Chinese HDTV chipset (League & Altobeam).

 

OTOH, could we capture the raw EPG signal from TVB and send it to DVBViewer's development/testing team?

[leeyc0]

If you emailed TVB's tech support, I believe the engineers there would just blame DVBViewer Pro. They would say TVB's EPG is working fine in hardware set-top boxes. So it's only the problem of DVBViewer, or maybe those Chinese HDTV chipset (League & Altobeam).

 

OTOH, could we capture the raw EPG signal from TVB and send it to DVBViewer's development/testing team?

 

I am already doing similar thing as hinted by griga, which seems to get the exact symptom (EPG event ID being reused). Although it is absolutely possible to have a short broadcast dump for them, I think I need to have a 1-2 minutes of the sample dump and repeat 2-3 days for them to reproduce. Since the source of the problem is already known by griga, I think it is optional to provide the dump for them, unless they requested it...

Edited June 10, 2012 by leeyc0

[Griga]

I guess the broadcaster changes the event ID assignment on each EPG update (typically at midnight), so the events A, B, C... get different IDs than before, or even worse, the ID previously assigned to event A now references event B or C.

 

A simple receiver that only displays the "just received" EPG will not be affected by this inconsistency. However, it is deadly for more sophisticated devices that increase the EPG availability by storing the data, or that try to keep track of an event over a longer period of time (e.g. recorders).

 

Though this "bad behaviour" is not explicitely prohibited by the DVB specifications, there are recommendations (e.g. from NorDig):

 

- An event ID, once assigned to an event, should not change. If an event is canceled or replaced by another one, its event ID should be dropped.

 

- An event ID should not be re-used within 90 days (which is no problem, because there are 65536 different ones).

 

There is one thing you can try: Untick Options -> EPG -> Persistent EPG Data in DVBViewer Pro, and delete the file epg.dat in the configuration folder in order to get rid of old data. This will prevent DVBViewer from storing the EPG. It will only be available for a channel after having received the channel / frequency during a DVBViewer session. However, the issue may still occur if the EPG is updated while DVBViewer is running.

[leeyc0]

griga: As I can see from broadcast dump, TVB simply sorted the EPG by time every day, and assigns every event starting from 1, so an event numbered #42 yesterday can be numbered #4 next day, and #42 refers to some other event... (You can now see how crappy the TV station is....)

 

I will try unticking the option "Persistent EPG Data" and see what's happening if I keep DVBViewer Recording Service running and let it update the EPG every 12 hours. As I got from your word "However, the issue may still occur if the EPG is updated while DVBViewer is running", it seems to me that the problem would still there if I keep the DVBViewer Recording Service running. If it is the case can you have a workaround for this problem (that is, simply forget all the EPG entries before every update if "Persistent EPG Data" is not ticked)?

Edited June 11, 2012 by leeyc0

[leeyc0]

@griga: Argh, the epg.dat is always created by DVBViewer Recorder Service. We badly need the same "Persistent EPG Data" option in DVBViewer Recorder Service. And I just said before, if this option is disabled, every EPG update by DVBViewer Recorder Service would simply throw away the original EPG data...

Edited June 11, 2012 by leeyc0

[Lars_MQ]

Disable EPG receiving for the recoding service (-> Do not receive EPG Data) all together, because it is of no use in such a scenario. Or disable it in the hardware options for each card.

 

While it may be acceptable for a receiver kind of device or programm to handle EPG the way @griga described it in the first part of his post, it does simply go against most of the "higher" functions of the RS.

 

You can use the basic functions, the server part and (limited) the recording part, but all EPG bound services will NOT work with the EPG provided by the broadcaster.

[leeyc0]

@Lars_MQ:

 

1. We need EPG for convenience of scheduled recording by EPG, so disabling EPG altogether is unfeasible. (And EPG is broken for only one TV station, the other TV station is working properly since it follows the guideline.) And it seems to me that all the EPG features are working properly, given that we reset RS every day....

2. Can you describe further how broken EPG would cause problem to RS "higher" function?

 

If it is a real trouble, we will stick to the old way of resetting recording service every day amd triggering an EPG scan to workaround this problem...

Edited June 11, 2012 by leeyc0

[leeyc0]

@Mr. Man-wai Chang: To be exact, this problem would occur when an EPG scan is triggered after TVB broadcasts new EPG data at around midnight, and RS has a copy of old EPG data.

[EDC]

I use "Recording Services API" and "wget" to solve the problem with TVB epg.

 

- Uncheck Automatic EPG Update every 12 hours if PC is active.

- Add an external task in Recording Services, e.g. Clean EPG,

- Point to wget.exe in your PC and set parameter "-q --spider http://<login>:<password>@<ip>:<port>/index.html?epg_clear=true"

- Schedule Clean EPG task and Scan EPG task in midnight.

[EDC]

Correct parameter and screen shot for external task.

 

- Point to wget.exe in your PC and set parameter "--quiet --spider http://<login>:<password>@<ip>:<port>/index.html?epg_clear=true" (without double quote)

 

Edited July 24, 2012 by EDC

[andyyau]

Thanks for EDC's help.

 

I have tried that and will see if it can fix the problem.

 

This is just a workaround, I do hope to fix it in DVBViewer.

 

I think in HK no one will expect TVB to fix this. It is a hopeless TV station.

[andyyau]

What is the best time for the timer to clean EPG? I set it to 5:00am and the EPG is collaped. Will 12:00am be to early?

 

And what is the best time to start rescan?

[EDC]

You may try to schedule it purge and re-scan every 12 hours, e.g. 00:01 and 12:01.

 

In case your EPG is corrupted, you can always login to web UI, go to "Tasks" and do re-scan manually.

[andyyau]

This method does not work any more recently. Once I clear and rescan EPG manually, TVB program information will get updated and it will disappear again very soon.

 

Any help?

[andyyau]

I also find the EPG information will disappear one by one. And finally all of the TVB program information will completely disappear.

[Paul]

I ever wrote a VB Script to clear and rescan the EPG, it runs two years so far no big problem, only sometimes DVBViewer main program display empty EPG when start, just restart it will be ok. You may take a look and try it http://digital.xy.hk/news/?postid=4096

[andyyau]

Hi Paul,

 

I tried the VBS script but got error when I double click it to run:

 

Line 56

Char 1

Error: Access is denied

 

Code 80070005

Source msxml3.dll

 

I am running it on Win7-64.

Edited August 12, 2012 by andyyau

[Paul]

Hi Andy,

 

Did you add 127.0.0.1 to the trust devices list?

Edited August 12, 2012 by Paul

[andyyau]

Hi Andy,

 

Did you add 127.0.0.1 to the trust devices list?

 

Do you mean to add inside svcuserdata.xml?

Yes I did but every time it is removed automatically after I run the script.

[majstang]

before editing config files stop RS-->edit the file-->save-->restart RS

[andyyau]

before editing config files stop RS-->edit the file-->save-->restart RS

 

The trust device is now saved and will not be removed.

Bit I still get the same error.....

 

 

How hard it is to use DVBViewer......

[Paul]

Hi Andy,

 

Please also try to add http://127.0.0.1 to Trust sites inside Internet Option.

[andyyau]

Paul,

 

Adding trusted sited to IE/Internet Option finally works.

 

But after running the script, there are two outcome:

1. The EPG data of current day is initially OK but will be clean after the end of the script. All EPG of the other days remains OK.

2. The EPG data of current day is OK but all the other days are empty.

 

So is it a requirement to set the starting time so that all of them are OK?

 

 

Thanks a lot!

[andyyau]

PS: Most of the time, it is condition 1 stated above. I cannot always re-produced condition 2.

[Paul]

What is the Time Zone setting?

[Paul]

Important!!

 

Seems the svcuserdata.xml does not store password anymore, you need to include the user password and guest password in the VB script at line #10 and #12, otherwise the VB script will clear your password after executed.

[andyyau]

What is the Time Zone setting?

 

 

auto time zone, Difference to GMT (minutes): 480

 

After around 12-18 hours, I find most EPG information is cleared, I now get only data on Friday to Tue.

Edited August 15, 2012 by andyyau

[andyyau]

Changed to manual time zone with value of 480 and adding user name and password to script.

It seems all OK now, but need to wait and see if EPG can be kept....

[Paul]

Auto time zone is not working since the first version I used two years before.

Edited August 16, 2012 by Paul

[andyyau]

After several hours, some of EPG info was still cleared. How can I stop EPG receiving completely?

[Paul]

The script turns off "Receive EPG Data" function every time after executed, please login to the Web Interface -> Configuration and uncheck "Automatic EPG Update every 12 hours", "Receive EPG Data" and "Receive MediaHighway EPG Data" if any of them is still enabled.

Edited August 17, 2012 by Paul

[andyyau]

I run the script yesterday. After end of the execution, I find the above selections are all unchecked in the config page.

But after 6-12 hours later, the EPG from today to sunday are all disappeared!

 

Is there a way to check the DVBViewer activity?

Edited August 18, 2012 by andyyau

[Paul]

I setup the Scheduler to runs this script every 12 hours, not sure whether this running schedule will help to solves your problem or not.

 

Please also check and confirm the "Timezone" is 480 and "AutoTimezone" is 0 in service.xml (although you see it on webinterface).

[andyyau]

Paul,

 

In the Configuration/Recorder section, I unselected all "EPG Info and Logging" and now it works for 2 days.

 

I hope the problem will be come back again.

 

Thanks!

[Delphi]

Maybe this can be of some help:

 

http://epgcollector.sourceforge.net/

 

I do not know the nuts and bolts of this app.

 

If the import cannot go to the Recording Service but the DVBViewer only, the Xepg should be able to do the job.

The Xepg will construct unique EventIDs.