Issues with overlapping EPG entries

[hugh2]

I see this relatively often, in both GE and Pro.

 

 

There are two issues.

 

(1) Two programs appear to overlap. (in this case Carry on Henry and the first Six Feet Under)

(2) When programs overlap and one is completely "contained" within the other, it is impossible to click on and select the shorter one.

 

(2) should be easy enough to fix. (1) may or may not be the "fault" of the TV station. It would be good to add a function to DVBViewer to clear the EPG for the current channel at the user's request, to allow it to repopulate with fresh data and see if that fixed overlapping programs like this. If it did, that would be evidence that DVBViewer's EPG update algorithm could be improved.

 

I used "fault" in quotes because at least one time I saw overlapping programs which seemed sort of legitimate. There was a kids' show lasting several hours which hosted various cartoons, and both the overall show and the individual cartoons were listed in the EPG - the latter contained in the former. Is "hierarchical" EPG data legitimate?

support.zip

[majstang]

...that would be evidence that DVBViewer's EPG update algorithm could be improved.

I have long suspected there is a bug in the DVBViewer EPG update function, when updating from RS EPG.dat. The EPG in RS do not show these overlappings, but DVBViewer EPG does.

Here's my thread about it over a year ago...pictures removed though but pretty much the same trouble.

http://www.DVBViewer.tv/forum/topic/44869-epg-problem-DVBViewer-4601/

[Griga]

Is "hierarchical" EPG data legitimate?

I've never seen it here in Germany, except in case of faulty EPG data. But it's not prohibited by the specs, as far as I know.

 

When programs overlap and one is completely "contained" within the other, it is impossible to click on and select the shorter one.

On selection DVBViewer searches through the chronologically ordered EPG data from left to right in order to find matching coordinates:

 

if (XMouse >= Entry.Left) and (XMouse < Entry.Right) then... //found, select it.

 

I wonder if searching in descending order from right to left (so the event with the latest start time is found first) would fix it...

[hugh2]

On selection DVBViewer searches through the chronologically ordered EPG data from left to right in order to find matching coordinates:

 

if (XMouse >= Entry.Left) and (XMouse < Entry.Right) then... //found, select it.

 

I wonder if searching in descending order from right to left (so the event with the latest start time is found first) would fix it...

 

Yes. They should be searched in the opposite order from how they are painted, so that a click would always select what appears "topmost" (in the z-order).

[Lars_MQ]

I believe it would be better to tell your broadcaster to use more common sense for their data... it doesn't matter which direction is choosen to iterate the data, as long as they send out ambiguous data there will be conflicts.

[hugh2]

I believe it would be better to tell your broadcaster to use more common sense for their data... it doesn't matter which direction is choosen to iterate the data, as long as they send out ambiguous data there will be conflicts.

I think they do so occasionally, and maybe that is legitimate, maybe not.

 

But... I just did an experiment which I think proves that DVBViewer is sometimes (most of the time?) at fault. I found an overlapping part in the EPG:

 

 

I then shut down DVBViewer Pro, deleted (actually renamed) epg.dat, started DVBViewer Pro again, tuned the channel and allowed a minute for the EPG to repopulate. It fixed the overlap:

 

 

My theory is that DVBViewer has an issue when the station changes their schedule and sometimes items from both the old and new versions of the schedule end up co-existing in DVBViewer's EPG.

 

Would you please at least consider implementing my idea of a function to clear EPG data for a channel?

[majstang]

My theory is that DVBViewer has an issue when the station changes their schedule and sometimes items from both the old and new versions of the schedule end up co-existing in DVBViewer's EPG.

Very intresting test hugh2 and it definitely proves DVBViewer EPG Update function has a bug. Your theory is spot on also. In my case the update function brought back a 3 month old show (not present in the current EPG.dat at the time) and overlapped it with the current EPG.dat, like if the Update function uses some kind of cache memory that wasn't cleared alright before populating the new EPG.dat. Old stuff is lurking around and gets mixed up with new stuff in the update. It resembles very much the Google Chrome browser cache problems where you have to hit the reload button sometimes more than once to clear the cache.

 

It is also intresting to know if the "MAD" show even exist in the current EPG.dat?

 

Regards

Majstang

Edited May 20, 2012 by majstang

[hugh2]

It is also intresting to know if the "MAD" show even exist in the current EPG.dat?

I don't think it does exist any more, although it is hard to be certain reading a proprietary file format. (i'm just opening it in Bintext)

[majstang]

I don't think it does exist any more, although it is hard to be certain reading a proprietary file format. (i'm just opening it in Bintext)

With the Recording Service you get the whole EPG.dat displayed in the webinterface and there you can search it. DVBViewer/EPG Window/TV Guide has a searchfunction as well. Well, your point is proven perfectly great without knowing the "MAD" show status anyways, so lets hope that is enough to make the developers find the bug

Edited May 19, 2012 by majstang

[Griga]

Yes. They should be searched in the opposite order

I hope it won't cause opposite issues.

 

In my case the update function brought back a 3 month old show

Certainly the broadcaster's fault, no DVBViewer bug.

 

My theory is that DVBViewer has an issue when the station changes their schedule and sometimes items from both the old and new versions of the schedule end up co-existing in DVBViewer's EPG.

Yes, it's difficult to handle. The broadcasted EPG data is not really designed for being stored in a data base like DVBViewer does, at least not for a longer period - there is no message saying "this Event ID is invalid now and shall be dropped". It just doesn't appear anymore, and extensive comparison would be required to detect it.

 

Would you please at least consider implementing my idea of a function to clear EPG data for a channel?

It rather should happen automatically. However, it's difficult again, because several parts of DVBViewer using EPG data don't copy it, but - for efficiency and low memory consumption - only hold a reference to it. Deleting entries without additional DVBViewer-wide measures would let pointers point to nowhere and cause access violations. Furthermore EPG data is received in the streaming thread, but displayed / handled in the UI thread, which are asynchronous, so each read / write / delete access requires a special handling with synchronisation objects like critical sections to avoid memory corruption.

 

That's why DVBViewer GE never deletes EPG data during runtime. If you let it run long enough - some months or so - it eventually will fill all available memory with EPG data. The only refresh happens when DVBViewer GE is launched, because it discards all entries from epg.dat that are earlier than now. Still to be solved... AFAIK DVBViewer Pro already contains some handling to avoid this 24/7 issue. I will have to ask Lars how he did it...

[hugh2]

...there is no message saying "this Event ID is invalid now and shall be dropped". It just doesn't appear anymore, and extensive comparison would be required to detect it.

Yes, perhaps it would be painful to detect events which were no longer being advertised in the EPG, but you could delete them based on other heuristics.

 

How about this: timestamp each event when you receive it from the EPG, and update its timestamp each subsequent time it is received, so the timestamps reflect recency not age. If two events overlap by more than some amount (say 20% of the shorter event), compare their timestamps. If they differ by more than a certain amount (say five minutes), the older one becomes a candidate for deletion.

 

my idea of a function to clear EPG data for a channel

It rather should happen automatically.

Agreed but what I was asking for as a quick and dirty workaround was something like a button the user could click to clear all EPG data for the current channel - and then let it re-populate.

However, it's difficult again, because several parts of DVBViewer using EPG data don't copy it, but - for efficiency and low memory consumption - only hold a reference to it. Deleting entries without additional DVBViewer-wide measures would let pointers point to nowhere and cause access violations. Furthermore EPG data is received in the streaming thread, but displayed / handled in the UI thread, which are asynchronous, so each read / write / delete access requires a special handling with synchronisation objects like critical sections to avoid memory corruption.

If actually freeing the objects is too dangerous (sometimes I wish all software was written in "safe" languages like C# / Java), just add a boolean field to indicate "deletion". No need for a synchronisation object for that.

[Derrick]

..just delete the epg.dat from time to time and start afresh

[majstang]

..just delete the epg.dat from time to time and start afresh

You're actually right, I tackled this issue by exactly doing that. Setting XEPG to "Delete old EPG prior to import" fixed it, haven't seen any overlappings ever since that (for over a year). Downside is the bug is still there though

[Griga]

just add a boolean field to indicate "deletion".

Not actually deleted, but "put aside" and marked as invalid? Not really smart The usual way of handling it is a reference count, as used by COM objects that are automagically deleted from memory as soon as the reference count gets zero.

 

However, the EPG may become a huge data collection, particularly if more than one satellite position is received. E.g. on Astra 28° East there is an EIT stream delivering the whole Freesat EPG for several transponders with 4.3 MBit/s, letting the TransEdit Analyzer collect 2,200,000 data items within 30 seconds, occupy 500 MB RAM (more than 1 GB before I optimised it), and the CPU load temporarily goes up to 100% on my poor old PC.

 

DVBViewer doesn't evaluate EPG data that exhaustively. Nevertheless efficiency and memory occupancy have to be considered. E.g. if there are 10,000 EPG entries, a single additional quadword data field (e.g. the arrival time) yields a total of 80,000 Bytes that have to be stored and processed.

[HaraldL]

..just delete the epg.dat from time to time and start afresh

 

I did this too already but after reading this I'm not sure if that can cause problems:

 

However, it's difficult again, because several parts of DVBViewer using EPG data don't copy it, but - for efficiency and low memory consumption - only hold a reference to it. Deleting entries without additional DVBViewer-wide measures would let pointers point to nowhere and cause access violations.

[hugh2]

Not actually deleted, but "put aside" and marked as invalid? Not really smart

It was not intended to be smart. I was merely responding to your concerns about implementation difficulty, with a quick-and-dirty method. It is still better than the current behaviour of keeping obsolete events which conflict with updated ones.

The usual way of handling it is a reference count, as used by COM objects that are automagically deleted from memory as soon as the reference count gets zero.

Implementation details are up to you. If I understood you correctly, you're currently not freeing EPG event objects at all. Which might be fine. I'm not asking that you free the objects, just stop displaying obsolete events in the GUI.

Nevertheless efficiency and memory occupancy have to be considered. E.g. if there are 10,000 EPG entries, a single additional quadword data field (e.g. the arrival time) yields a total of 80,000 Bytes that have to be stored and processed.

80 KB doesn't sound like so much. And each event has several string fields already, so why worry about an extra quadword?

[Griga]

It was not intended to be smart.

Well, it's simply out of the question, because it doesn't solve the basic problem, but introduces another issue (intended memory leak by user interaction, so to say). IMO any runtime EPG refresh function must be based on a more flexible data handling allowing to remove entries without having to take care of the consequences througout the whole DVBViewer code. That must be elaborated first. Adding functions without a conclusive concept and code base behind it just messes the software up.

 

I've already started working on it, but it will take time, because there are several aspects to consider, some of them already mentioned here, and there are a lot of other things to do. The cost-benefit ratio will be decisive...

[hugh2]

I've already started working on it

That's great. Will the work find its way into DVBViewer Pro?

The cost-benefit ratio will be decisive...

I think it will make a lot of users happier

[nars]

Was there any progress on this? I have not been running the very latest version as I have been very busy on last months and didn't care about checking updates / updating it (also it was working fine, except this 'small' issue then I didn't bothered to update), looking at changelogs there was no change on this but please pardon me if you actually did something to improve this, let me know and I will update and keep an eye on it again with latest version to see if it still happens.

 

I very very often notice this problem for a long time now, i.e. since I started using DVBViewer... 3 or 4 years... the problem happens to me when broadcaster do "last day" big changes on epg, like replacing some epg entry for something different and with different start/stop times... you see what I mean... DVBViewer ends leaving the old entry merged with the new entry after broadcaster update and showing both overlapped.

 

I understand that it may be crap from broadcaster doing such "last day" changes, however it happens... and sometimes epg becomes a real mess of old/new entries. The workaround I did found was to disable 'persistent epg data' option on DVBViewer options and when it happens, or when I want to be sure I'm seeing updated epg without any old entry merged, I do just close DVBViewer and re-run it, and because 'persistent epg data' option is disabled then I surely get fresh/updated epg when tunning the channel again after DVBViewer restart, it works ok, however this is a bit of a pain/nosense having to restart DVBViewer to refresh/reset epg... then I would also +1 suggest same that was already suggested on this topic, please if possible add an option to reset/clean epg data for a single tv channel, having it on context menu on epg window would be perfect IMO.

 

Thanks in advance.

Edited June 15, 2013 by nars

[Tjod]

In the RS you can activate a tweak "Clear the EPG List before an EPG update"

 

Some broadcasters mess up their EPG data, which causes problems with the long term handling in the RS. This setting clears all EPG Data before an EPG Update is started.

 

[nars]

In the RS you can activate a tweak "Clear the EPG List before an EPG update"

 

 

I don't really need RS, any chance such option could also be added on main DVBViewer (or the manual option...) ?

[Tjod]

I don't think that the DVBViewer Pro get a EPG auto update capacity. And without that, this option is pointless.

[nars]

Then manual clear epg function for specific channel would be the best... what is completely nosense is needing to close/re-run the app to get non messed epg

[Griga]

Was there any progress on this?

 

Yes, in DVBViewer GE. Download it from the members area, read the ReadMe and give it a try. Please note that it doesn't support decrypting with a CI/CAM (in case you need it). Encrypted channels can only be watched/recorded if decryption is done by the Recording Service and DVBViewer GE is working as a client.

 

I've implemented a new handling of inconsistent EPG data in DVBViewer GE that seems to be more robust, particularly if broadcasters show the bad habit of reusing Event IDs for different events on EPG updates. Additionally there is a EPG refresh mechanism that flags already stored EPG data as "old" on the following occasions:

 

- when DVBViewer GE is launched resp. the data is loaded from the file epg.dat

- when DVBViewer GE wakes up from standby / hibernate

- every hour while DVBViewer GE is running

 

Old EPG data is not discarded, but remains stored. However, incoming new data can displace overlapping data if (and only if) it is flagged as old. This method doesn't guarantee that the EPG is always clean, but a mess of old and new entries should disappear after some time (if the EPG is updated while DVBViewer GE is running) and should automatically be cleared when DVBViewer GE is launched or wakes up from hibernate / standby.

 

Unfortunately DVBViewer can't replace previously received overlapping data immediately, because overlapping is allowed by the specifications, e.g. for news embedded in some other programme (see beginning of our discussion). So overlapping data must at least get a chance to coexist.

 

BTW: The problem of EPG data deletion during runtime (mentioned here) is solved, which means, the EPG refresh mechanism periodically deletes all entries that are older than 36 hours, so a DVBViewer GE instance running 24/7 won't gradually eat up all memory. As a consequence a manual "Clear a Channel's EPG" function would be possible now. However, I think it is better to let it happen silently in the background.

 

The new handling has to pass more on-road tests, though. One big problem is I can't receive channels here that are doing such nasty things with the EPG. I've already tried a kind of simulation by adding code that purposely messed the data up (e.g. by shifting all events one hour to the future), and the results were quite promising. However, "in the wild" is a different thing... let me know how it works.

[nars]

Thank you Griga, I will try to test it and let you know.

 

Btw, if I provide you a copy of epg.dat of when the overlaping mess happens would that help you in any way to diagnose it?

[nars]

@Griga: I did tried GE some times but haven't been able to reproduce the problem on it until now, not sure if "luck" or... but I did returned to use the DVBViewer version I was using before, and today I did reproduced it. I'm attaching a copy of my epg.dat and a screenshot showing the overlaping (note the date/time on the file was the one when screenshot was taken), not sure if this can be of any help for you to see the problem?

epg_dat.rar

Edited July 21, 2013 by nars