IPTV EPG in Neutrino CL (collaboration for plugin development)

[leviscar]

Hi All,
I have used and currently still use Neutrino CL (CL=Channel List) plugin:
http://www.DVBViewer.tv/forum/topic/42918-customized-programmliste-neutrinocl-plugin/

 

Surely many users have used and appreciated it.
But I would report the same a brief description of this plug-in, in order 
to highlight its great usefulness.  

Neutrino CL is an add-on supported by many osd interfaces, with the following key features:

 

- It shows your channel list, organized in different sub-lists. 
Each sub-list corresponds to a folder that you have previously defined in the channel editor.
When the neutrino CL is displayed you can simply press left and rigth keys
to switch between different sub-lists.
Up and down keys are used to scroll, and enter to select the channel.

 

- EPG information are also showed in a very synthetic way. At the righ of each channel name
you can immediately see which is the current program broadcasted on that channel.

 

I will had some plugin snapshots as next post. 

 

Unfortunately the plugin is not yet mantained by its creator, Pmneo.
Last revision date is January 14, 2013.  
But the good news is that the creator, kindly shared the source code for possible
future development by other people. 

 

The most important current limitation (due to the fact that the plug-in has not been updated since almost 4 years)
is that the IPTV channels are not fully supported: IPTV channels list are recognized and displayed,
but the EPG information is not correctly managed and displayed for them.

 

I do not have any experience on DVBViewer plugin development, neither on Delphi language.
But I have experience on other programming languages. 
So I gave a look at the Plugin Delphi code looking for the possible reasons why IPTV EPG information
is not correctly imported and displayed.
The main plugin source code file seems to be 'UPlugin.pas'. 
Another important source file is 'plgGlobals.pas' containg data and function definitions.
Looking for the string 'tuner' I noticed that the 'TS stream' tuner type (used for the IPTV channels)
is not supported.
I also notice that the EPG information is extracted form the DVBViewer database using the channel 
SID. This is another problem for the IPTV channel, because they have a dummy SID parameter,
often set to 1 for all of them (it should be instead be univocal, as happens for DVB-T/S channels).

 

So it seems that in order to support EPG data for IPTV channel, two main  modification
are needed:
1)extend the recognized 'tuner' types, by adding the 'TS stream' category;
2)don't use the SID as acces key to retrieve the EPG data, but another channel univocal parameter.  

 

Is there someone expert in plugin developing who could provide help 
to upgrade this plugin?
I think it could be very useful and appreciated to all the users who still are using this
very nice and powerful plugin. 
I do not see any other equivalent plugin with the same unique and very handy features...

[leviscar]

Here following you can see a DVB-T sublist with the correct EPG information:  

 

This is instead an IPTV sub-list. You can notice that the EPG info are missing: 

[Griga]

14 hours ago, leviscar said:

But the good news is that the creator, kindly shared the source code for possible
future development by other people. 

 

Where is it available? Maybe @Delphi can have a look at it. He is an EPG expert (see here), already implemented TS Stream support in his tool and is using Delphi for development, as his name suggests.

 

[leviscar]

Delphi source code is available here: http://www.DVBViewer.tv/forum/applications/core/interface/file/attachment.php?id=34090

If @Delphi could give an help it would be very appreciated :-)

 

[leviscar]

I have successfully compiled the source code, using Embarcadero DX 10.2.

In order to correctly compile I needed to replace "RegExpr.pas" with a more recent version downloaded from Internet.

Attached is the update archive.
@Delphi, this could be used as starting point....

neutrinoCL_source_embarcadero.zip

Edited December 12, 2017 by leviscar

[Delphi]

I had a short look into the code.

 

First of all you must extract a new version of the DVBViewerServer_TLB.pas from within the Delphi IDE, or I can ship you one.

 

Here is some of the code from Xepg. I use Delphi XE2.

 

From my UGlobals.pas:

 

type
  TEPGChannelID = record
    case Integer of
      0: (ID: Int64);
      1: (Lo, Hi: DWord);
      2: (SID: Word; TID: Word; NID: Word; TunerType: Byte; Source: Byte);
  end; // Source used internanny by DVBV/DMS, set to 0


  PEPGChannelID = ^TEPGChannelID;

  TStreamHash = record //64 bit ID for TS Stream
    case Integer of
      0: (ID: Int64);
      1: (IDLo, IDHi: DWord);
      2: (IDLoLo, IDLoHi, //Symbolrate
          IDHiLo, IDHiHi: Word); //LOF, DiSEqCValue
    end;



//  PDVBVChannel = ^TDVBVChannel; //old style
  TDVBVChannel = class //record
    rEPGChannelID: TEPGChannelID;
    ChannelNo: Integer;
    IsEncrypted, IsFavorite, IsRadio: boolean;
    Root, Name, Category: TChannelName;
  end;
  PDVBVChannel = TDVBVChannel; //I was lazy



var
  DVBVChannels: TList = nil; // of TDVBVChannel

const
  ttCable = 0;
  ttSatellite = 1;
  ttTerrestrial = 2;
  ttATSC = 3;
  ttIPTV = 4;
  ttStream = 5;
  ttFile = 6;  //TS Stream

 

Here is how I read the channel list (simplified):

 

procedure TAbstractThread.GetDVBVChannels;
var
  pChannelCollection: IChannelCollection;
  pFavoritesManager: IFavoritesManager;
  List: OleVariant;
  i, N: integer;
  p: TDVBVChannel;
  trEPGChannelID: TEPGChannelID;
  TunerType: Byte;
  URLID: TStreamHash;

  function NewChannel(aEPGChannelID: TEPGChannelID): boolean;
  var
    i: integer;
    p: PDVBVChannel;
  begin
    Result := True;
    for i := 0 to DVBVChannels.Count - 1 do
    begin
      p := DVBVChannels[i];
      if (p.rEPGChannelID.ID = aEPGChannelID.ID) then
      begin
        Result := False;
        Exit;
      end;
    end;
  end;


begin  //GetDVBVChannels
  try
    pChannelCollection := FDVBViewer.ChannelManager;
    pFavoritesManager := FDVBViewer.FavoritesManager;
    N := pChannelCollection.GetChannelList(List);

    for i := 0  to N - 1 do
    begin
      TunerType  := List[i, 4];
      if TunerType = ttFile then
      begin
        URLID.IDLo := List[i, 6]; //Symbolrate
        URLID.IDHiLo := List[i, 7]; //LOF
        trEPGChannelID.SID := Max(URLID.IDLoLo, 1);
        trEPGChannelID.TID := Max(URLID.IDLoHi, 1);
        trEPGChannelID.NID := Max(URLID.IDHiLo, 1);
      end else
      begin
        trEPGChannelID.SID := List[i, 26];
        trEPGChannelID.TID := List[i, 23];
        trEPGChannelID.NID := List[i, 25];
      end;
      trEPGChannelID.TunerType := TunerType + 1;
      trEPGChannelID.Source := 0;

      if NewChannel(trEPGChannelID) and
       ((List[i, 22] <> 0) or (List[i, 20] <> 0)) then //Vidio or Audio PID <> 0
      begin
        p := TDVBVChannel.Create;
        DVBVChannels.Add(p);
        with p do
        begin
          rEPGChannelID := trEPGChannelID;
          Root := List[i, 0];
          IsRadio := List[i, 22] = 0; //Video PID
          Category := List[i, 2];
          Name := List[i, 1];
          IsEncrypted := List[i, 3] and 1 <> 0;
          ChannelNo := i;
        end;  // with p
      end;
      if Terminated Then Exit;
    end; //for i
  except
    if ThreadError = 0 then ThreadError := errReadingChannels;
    raise;
  end;
end;

 

For the TS Stream a hash value is used instead of NID-TID-SID. This solves the EPG/IPTV problem. You must store the TEPGChannelID for each channel and use that for fetching the EPG.

 

I think (have never used them to read EPG) it is IEPGManager2(disp) in DVBViewerServer_TLB.pas that can come in play for reading the EPG. Be aware that not all COM funtions are upgraded to use hash values.
 

Edited December 12, 2017 by Delphi

[leviscar]

Many thanks for your detailed explanation. I will go deeper and come back to you with possible questions ;-)

Can you please share the new version of "DVBViewerServer_TLB.pas"?

 

A first question: after successfully re-compiling with Embarcadero DX, I obtained a final .dll file which is bigger than the original file  (2.2 MB instead of 450KB).
When I replace the old .dll file with the newer one and run DVBViewer, the plugin is not loaded. Is there a way to debug the reason why the new plugin is no longer recognized? 

Edited December 12, 2017 by leviscar

[Delphi]

The string type in Delhi 2009 and later is a UTF-16 encoded one. DVBViewer expects AnsiString. Use type AnsiString instead of type string when registering the plugin .dll.

[leviscar]

Could you please explain how to do that? Should I modify the .dpr file? Sorry for this basic question, but I have no experience on DLL development.

[Delphi]

Yes, in neutrinoCL.dpr:

 

function Version: Pchar; stdcall;
begin
  result := PChar(NeutrinoCL_Version);
end;

function Copyright: PChar; stdcall;
begin
  result := 'Philip Mair';
end;

function LibTyp: PChar; stdcall;
begin
  result := 'Plugin';
end;

function PluginName: PChar; stdcall;
begin
  result := 'neutrinoCL';
end;

 

Change PChar to PAnsiChar.

 

Note1: You should convert your IPTV channels (Tunertype=4) to TS Streams (Tunertype=6) as described here:

 

http://www.DVBViewer.tv/forum/topic/59663-xepg/?do=findComment&comment=465399

 

Note2: When using the COM interface all strings are WideString (UTF-16), so is the basic string type in Delphi 10.2.

 

EDIT: You can debug a .dll by (Delphi XE2): Run -> Parameters... -> Host Application. Browse to DVBViewer.
 

Edited December 13, 2017 by Delphi

[Delphi]

I have been looking a bit more into the code. You need an update to the TTuner declaration in plgGlobals.pas. Attached.

 

Most important is that I have added a function ChannelID. I think it is worth a try to use that in 

 

IEPGManager2 -> GetAsArray2 in the DVBViewerServer_TLB.pas instead of

 

    ci := getSelectedChannel;
    epg := DVBViewer.EPGManager.Get(ci.channel.Tuner.SID, ci.channel.Tuner.TransportStreamID,Now, Now + 30);

in UPlugin.pas.

 

Uupdate.zip

 

EDIT: In members area the is some documentation of the format of channels.dat explaining the TTuner. Not quite uptodate though.

 

Edited December 13, 2017 by Delphi

[leviscar]

Thanks for this additional suggestion. I have successfully compiled the code as currently is, and the resulting DLL is correctly imported by DVBViewer now. This is an important preliminary step :-) Now I'm ready to try updating the code in order to correctly import EPG data for IPTV channels. I will start by following your suggestion ;-)

[Griga]

14 hours ago, Delphi said:

Most important is that I have added a function ChannelID. I think it is worth a try to use that in 

 

IEPGManager2 -> GetAsArray2 in the DVBViewerServer_TLB.pas instead of

 

I had a look at it and compared it with the DVBViewer code. That's the way to go. The result of function TTuner.EPGChannelID (provided by @Delphi) must be passed to IEPGManager.GetAsArray2 in order to get the imported EPG for a TS Stream channel.

 

TTuner.EPGChannelID returns an EPG Channel ID for TS Stream channels that is not based on the Service ID, Transportstream ID and Network ID (which are ambiguous for IPTV), but an EPG Channel ID that is derived from the MD5 hash of the URL, so that IPTV channels with a different URL yield a different ID.

 

Please note that this will only work with EPG data originating from an importer like Xepg that supports this kind of hash derived EPG Channel ID. It will not work with the native EPG of IPTV channels (if there is any). A native EPG that is embedded as Event Information Table (EIT) in the received transport stream can only be obtained by using a SID/TID/NID derived EPG Channel ID.

[Delphi]

@Griga:

 

The declaration of GetAsArrray2 is

 

    function GetAsArray2(ChannelID: Int64; StartTime: TDateTime; EndTime: TDateTime; 
                         out List: OleVariant): Integer; safecall;

That's why I suggested to use TTuner.ChannelID, but you suggest to use TTuner.EPGChannelID. Which one is the right one? The latter is the most logical one in spite of the declaration.

 

@leviscar:

 

The declarations in the Uupdate.pas are meant as an inspiration only.The TTuner as delivered reflects how things are stored on disk (channels.dat). You can remark things out you don't need (Ctrl+'). Furthermore, I recommend to change

 

TChannelStr = string[25];

to just

  TChannelStr = string;

 

 

 

Edited December 14, 2017 by Delphi

[Griga]

32 minutes ago, Delphi said:

That's why I suggested to use TTuner.ChannelID, but you suggest to use TTuner.EPGChannelID. Which one is the right one?

 

It must be TTuner.EPGChannelID.

 

35 minutes ago, Delphi said:

Furthermore, I recommend to change

 

The declaration is intended for direct channels.dat access! Therefore TChannelStr must be a fixed size string with one length byte and 25 character bytes, not a dynamic variable length string. It must match the the string as stored in the channels.dat.

[leviscar]

TNeutrinoCL = class(TBaseOSDPlugin)
...
    dvbViewer2: IDVBViewer2;
...

var
epg2 : Integer;
id2 : Int64;
epg_data: OleVariant;
...

id2 :=  ci.channel.Tuner.EPGChannelID(); //calling function in this way does not work!
epg2 := dvbViewer2.EPGManager2.GetAsArray2(id2, Now(), Now()+30, epg_data);

I have started my implementation as showed above.

I have two questions.

 

1) The instruction "ci.channel.Tuner.EPGChannelID();" does not work as I would expect. ('EPGChannelID' results as not defined).
....Probably I got the reason of this. When I write "ci.channel.tuner" I obtain the tuner property as defined in the 'DVBViewerServer_TLB.pas' file.
I did not obtain the new custom TTuner type as defined in 'plgGlobals.pas', as I would expect .
How to point to this custom tuner type?

 

2) Could you please provide more details about the format of the GetAsArray2 output parameter?
It is an OleVariant type. I named it "epg_data".

 

Many thanks!

Edited December 14, 2017 by leviscar

[leviscar]

 

Edited December 14, 2017 by leviscar

[Delphi]

8 minutes ago, leviscar said:

1) "ci.channel.Tuner.EPGChannelID();" is not recognized as a correct syntax. Which is the correct one?

 

Remove the (), try

 

id2 := ci.channel.Tuner.EPGChannelID;

 

13 minutes ago, leviscar said:

2) Could you please provide more details about the format of the GetAsArray2 output parameter?
It is an OleVariant type. I named it "epg_data".

 

 

I don't know. Maybe @Griga will document.

 

26 minutes ago, Griga said:

The declaration is intended for direct channels.dat access! Therefore TChannelStr must be a fixed size string with one length byte and 25 character bytes, not a dynamic variable length string. It must match the the string as stored in the channels.dat.

 

I guess you might be right in this context (plugin). However, I don't see any direct channels.dat access in the code. It's 10 years since I last worked with plugins. I may be wrong.  Maybe it's just a bad idea to use the name TTuner? I will leave the decision to @leviscar.

[Delphi]

Just an idea. Maybe you should create your own channellist consisting of TTuner records using ICollection as in Xepg,

 

http://www.DVBViewer.tv/forum/topic/60910-iptv-epg-in-neutrino-cl-collaboration-for-plugin-development/?do=findComment&comment=469742

 

The definitions are:

 

    List[u, 0] := achan.Root;
    List[u, 1] := achan.wName;
    List[u, 2] := achan.Category;
    List[u, 3] := achan.tuner.flags;
    List[u, 4] := integer(achan.Tuner.TunerType);
    List[u, 5] := achan.Tuner.Frequency;
    List[u, 6] := achan.Tuner.SymbolRate;
    List[u, 7] := achan.Tuner.LOF;
    List[u, 8] := achan.Tuner.PMT;
    List[u, 9] := achan.Tuner.Volume;
    List[u, 10] := achan.Tuner.SatModulation;
    List[u, 11] := achan.Tuner.AVFormat;
    List[u, 12] := achan.Tuner.FEC;
    List[u, 13] := achan.Tuner.Unused3;
    List[u, 14] := achan.Tuner.Polarity;
    List[u, 15] := achan.Tuner.OrbitalPos;
    List[u, 16] := achan.Tuner.Tone;
    List[u, 17] := achan.Tuner.DiSEqCValue;
    List[u, 18] := achan.Tuner.Diseqc;
    List[u, 19] := string(achan.Tuner.Language);
    List[u, 20] := achan.Tuner.AudioPID;
    List[u, 21] := achan.Tuner.EPGFlag;
    List[u, 22] := achan.Tuner.VideoPID;
    List[u, 23] := achan.Tuner.TransportStreamID;
    List[u, 24] := achan.Tuner.telePID;
    List[u, 25] := achan.Tuner.NetworkID;
    List[u, 26] := achan.Tuner.SID;
    List[u, 27] := achan.Tuner.PCRPID;
    List[u, 28] := Ord(achan.Tuner.Group);

Then you can have type TChannelStr = string;

 

Another possibility (EDIT: the best I think)  is to use the ITuner interface from DVBViewerServer_TLB.pas and then calculate the EPGChannelID yourself inspired by the code in Uupdate.pas.

 

Edited December 14, 2017 by Delphi

[Delphi]

The following could be useful:

 

function GetEPGChannelID(Tuner: ITuner): int64;
begin
  if Tuner.TunerType >= 6 then
  begin
    TEPGChannelID(Result).SID := Max(LongRec(Tuner.Symbolrate).Lo, 1);
    TEPGChannelID(Result).TID := Max(LongRec(Tuner.Symbolrate).Hi, 1);
    TEPGChannelID(Result).NID := Max(Tuner.LNB, 1); //I think this is right
  end
  else begin
    TEPGChannelID(Result).SID := Tuner.SID;
    TEPGChannelID(Result).TID := Tuner.TransportstreamID;
    TEPGChannelID(Result).NID := Tuner.NetworkID;
  end;
  TEPGChannelID(Result).TunerType := Tuner.TunerType + 1;
  TEPGChannelID(Result).Source := 0;
end;

Just use

 

id2 := GetEPGChannelID(ci.Channel.Tuner);

 

[leviscar]

Perfect. This seems to be fine. And I compiled it without any error.
(EDIT: I noticed you replaced the non-existent LOF field with LNB. I agree because LOF does not exist in ITuner).

 

Now I need to know the format of the following 'epg_data' variable (the output parameter of EPGManager2.GetAsArray2):

epg_data: OleVariant;
epg2 := dvbViewer2.EPGManager2.GetAsArray2(id2, Now(), Now()+30, epg_data);

in order to correctly extract the data that i need.
@Griga, could you please give an help on this?

[leviscar]

post deleted

Edited December 14, 2017 by leviscar

[Delphi]

To access the dvbViewer2 interface you will need to query for it, something like:

var
  N: integer;
  DVBViewer2: IDVBViewer2;
  Unknown: IUnknown;
...
    if not (GetActiveObject(CLASS_DVBViewer, nil, Unknown) = MK_E_UNAVAILABLE)
      then Unknown.QueryInterface(IID_IDVBViewer2, DVBViewer2);



    if Not Assigned(dvbViewer2) then
      Exit;



    id2 := GetEPGChannelID(ci.Channel.Tuner);



    N := dvbViewer2.EPGManager2.GetAsArray2(id2, Now(), Now()+30, epg_data);

You will need System.ActiveX in the uses clause as well.
 

[Griga]

13 hours ago, leviscar said:

epg2 := dvbViewer2.EPGManager2.GetAsArray2(id2, Now(), Now()+30, epg_data);

 

This will yield all EPG items for the channel specified by id2 from now until now + 30 days (as far as available). For only retrieving what is running now it should be

var
  TimeNow: TDateTime;
  NrOfReturnedItems: Integer;
  epg_data: OleVariant;

TimeNow := Now;
NrOfReturnedItems := dvbViewer2.EPGManager2.GetAsArray2(id2, TimeNow, TimeNow, epg_data);

//or for getting the "running now" epg items of all channels at once:

NrOfReturnedItems := dvbViewer2.EPGManager2.GetAsArray2(0, TimeNow, TimeNow, epg_data);

The EPGManager.GetAsArray2 handler in DVBViewer creates a list of matching EPG items and stores them in epg_data as follows:

  for i := 0 to EPGItemList.Count - 1 do
  begin
    epg_data[i, 0] := EPGItemList[i].EPGChannelID; //EPG Channel ID as Int64
    epg_data[i, 1] := EPGItemList[i].EventID; //Event ID as unsigned Integer (DWord)
    epg_data[i, 2] := EPGItemList[i].Time; //start time as TDateTime
    epg_data[i, 3] := EPGItemList[i].Duration; //duration as TDateTime
    epg_data[i, 4] := EPGItemList[i].EventW; //subheading as UTF-16 string
    epg_data[i, 5] := EPGItemList[i].TitleW; //title as UTF-16 string 
    epg_data[i, 6] := EPGItemList[i].DescriptionW; //long description as UTF-16 string
    epg_data[i, 7] := EPGItemList[i].CharSet; //character set of the received data as byte (can be ignored)
    epg_data[i, 8] := EPGItemList[i].Content; //program content as byte (see ETSI ETSI EN 300 468 6.2.9, Content Descriptor)
  end;
  result := EPGItemList.Count;

 

[Delphi]

I have a smarter way to query for the IDVBViewer2 interface:

implementation
uses PlgGlobals, RegExpr, Variants;

// add the following code to uPlugin.pas right after the implementation, uses section
var
  FDVBViewer2: IDVBViewer2 = nil;

function DVBViewer2: IDVBViewer2;
var
  Unknown: IUnknown;
begin
  if not Assigned(FDVBViewer2) then
    if not(GetActiveObject(CLASS_DVBViewer, nil, Unknown) = MK_E_UNAVAILABLE) then
      Unknown.QueryInterface(IID_IDVBViewer2, FDVBViewer2);
  Result := FDVBViewer2
end;

If you need to access DVBViewer2, just do it. Connection will be etablished automatically.

 

In procedure TNeutrinoCL.updateEPG(); you must delete
 

var
  DVBViewer2:  IDVBViewer2;

if present.

[leviscar]

The updated plugin is finally nearly to be completed. Thanks to your suggestions I implemented the reading of the EPG data reading by using the EPGmanager2 API.
I have successfully tested it: now also IPTV EPG data are correctly imported :-). I will share the new plugin after updating also some secondary procedures.
I would also implement a better way to manage the channel logos: the plugin currently looks for logo image files with the same name as the channel name,

ignoring the logos defined using the dedicated DVBViewer option tab.
Is there a way to read the logos directly from the DVBViewer internal database?

[Griga]

Have a look at the configuration folder -> ChannelLogos2.ini that contains EPGChannelID=logo filename assignments.

 

However, DVBViewer creates these assignments on demand (by using the Ratcliff similarity algorithm) when they are needed, so you can't expect the list to be complete.

[leviscar]

So, if I associate a logo to a channel using the DVBViewer Options/Channel_logos window, will the ChannelLogos2.ini file be update with my new defined associations? Moreover, is there a library function which returns the ChannelLogos2.ini path?

[Delphi]

2 hours ago, leviscar said:

Moreover, is there a library function which returns the ChannelLogos2.ini path?

 

Yes:

  IDVBOSD = interface(IDispatch)
    ['{CD78F42E-9DBA-459E-B1B9-8747CCE9318D}']

   ...
    function SettingDir: WideString; safecall;

 

[Griga]

3 hours ago, leviscar said:

will the ChannelLogos2.ini file be update with my new defined associations?

 

Yes, sooner or later. Dunno if DVBViewer does it immediately or must be closed first. Try....

[Delphi]

Since you allready have an initialized DVBViewer variable, you can just do

var
  ConfigFolder: TFileName; //or just string
...
   ConfigFolder :=  DVBViewer.OSD.SettingDir;

 

[Delphi]

On 15/12/2017 at 11:20 PM, leviscar said:

now also IPTV EPG data are correctly imported :-)

 

Great that you made it work. Just out of curiosity, I have a few questions:

 

1) Did you just convert your IPTV channels (Tunertype=4) to TS Streams (Tunertype=6) the way I suggested?

2) Do you use external EPG (e.g. Xepg) to get EPG for the converted TS Stream channels?

[leviscar]

1) The IPTV channels in my channel list were already defined as TS streams (tuner type=6). So, no conversion was needed. 

2) Yes, I use Xepg to correctly retrieve EPG data for TS Streams channels.

 

Attached is the source code file that I have changed.

I have also tried to attach a zip file containing the updated plugin .dll, but it is larger than the maximum size I'm authorized to upload (61 KB only). 

UPlugin.zip