IrpTransmogrifier: new program/library for IRP protocols

[Barf]

[Background: It is normal, and not an error at all, that an IR signal has more than one decode. A decode simply means that the signal fulfills certain criterions, and these does not need to be mutually exclusive. (Does "A" denote a Latin uppercase "a" or an uppercase Greek alpha? Is "table" a piece of furniture or a two-dimensional array?) However, it is usually preferable to have some sort of priority system, so that, at least in most cases, one decode is delivered. In DecodeIR, these rules are hard coded in the C++ code. This makes it not only very hard to change, but also very hard for the user to understand the exact rules in effect. In contrast, IrpTransmogrifier has in its data base (IrpProtocols.xml), elements that govern when a decode "triggers", and that certain decodes are preferred over others, the latter governed by the "prefer-over" parameter(s). (Examples where given in previous posts.)]

I have now uploaded build 5 of RMIR v2.09 to the RMIR Development folder. This incorporates IrpTransmogrifier 1.2.0 ...

Wow... that is 13 hours including the night hours...

Any reason for using an element rm:commentItem instead of irp:documentation?

Yes, it is not intended as documentation. It is a brief comment that RMIR includes in the Misc column on the Learned Signals tab, similar to corresponding text that DecodeIR provides. If you look at the learned signals in the new Test file 6, you will see this comment showing for the Sony DSP signals.

Ok, makes sense. But feel free to include an irp:documentation too

* G.I.4DTV needs prefer-over G.I.4DTVold.

Thank for the observation; more likely the latter should be removed instead.

* You still have NEC1 prefer-over Pioneer, is this is an oversight?

No, since NEC1 has <irp:parameter name="reject-repeatless">true</irp:parameter> (semantic should be obvious). If it has a NEC1-type ditto, then it is not a Pioneer.

I am still confused about Pioneer versus NEC2. I have a constructed signal, learned signal 157 in test file 1, that gives a multiple decode of NEC2 and Pioneer. It shows frequency 39801, D=90, S=165, OBC=38. I can't remember why I constructed this, but the D=90 appears to show that it isn't Pioneer. I just offer this for consideration, as it isn't a real signal but the multiple decode is not what I would have expected.

If the signal is approximately 40kHz, I would always assume this to be a Pioneer signal over an NEC2 signal. NEC1 and NEC2 are both 38kHz.

See above; multiple decodes are as such not an error IMHO. There are many parameters than can be tweaked, new in 1.2.0 are frequency_low and frequency_high. Suggestions are welcome.

Graham, I intend to merge you additions as soon as things have stabilized. Also, I intend to implement setting up an IrpDatabase from several files, normally one "official", and one or more "patchfiles", the latter complementing the first, possibly overwriting.

[Barf]

So, I have finally tested the program and made my way through Graham's document.

First of all, a big CONGRATULATIONS for getting this to work! I have not tried it extensively, but what I tried worked. Nothing that I say in the sequel should change this genuinely positive assessment.

It is quite a cryptic notation, and reading the 11 page document gives me a unpleasant feeling of unnecessarily complicate character manipulation.

I think it it wrong to characterize the work as an extension of the IRP notation ("Appendix to IRP notation"). It is another DSL (domain specific language) that builds upon the IRP, but it does not form, for example, a superset. (Also, you are implicitly crucially using extensions from IrpTransmogrifier (documented here) without reference.)

Short summary: The "executor descriptor" is associated with a single protocol, consists of a single text string, consisting of a few different parts, divided by a soup of parenthesis, brackets, braces, semicolons and colons. It describes how the current protocol can be applied to particular executor.

First there is the protocol ID, possibly followed by its version ("qulified id"). Then follows (optionally) the executor name ("Protocol" in RM), an optional predicate that allows the use only in certain situations, and a "sequencer" which effectively restricts the domain of the valid inputs (so it should really go to the parameterspec) -- "infospec". The next component "definitionspec" is nothing but the definitions from the IRP notation: a (possibly empty) set of assignents of expressions to names. Finally, the most important part, the "parameterspec", effectively mapping the protocol parameter to the executor parameters. Due to the nature of the problem, this necessitates some quite "hairy" wizardry in the form of "selectors" and "logical-or" expressions.

Alternatively, these parts could have been packed into XML elements and attributes, avoiding the parenthesis-bracket-brace-semicolon... magic.

The most crucial -- and hardest -- part is the parameterspec. It can however be left out, in which case special rules apply (Section 5.5). These rules are somewhat independent and "orthogonal" to the rest of the rules. I would here suggest just removing the entire section 5.5, and making the parameter spec mandatory. This would make the whole package easier to understand and use. (Ease of use is not the number of characters the user needs to type, rather it is the number of characters the author needs to explain to the user...)

Please allow, and use, longer, more descriptive variable names than the present one-character names. For example, in the Sony15 example, "force_sony15" would be a much more readable name than the present "X". (Possibly you want to use the class org.harctoolbox.irp.Name?)

The selector syntax is really "somewhat" hard to digest. Possibly disallowing the un-named selectors and writing D?A=2 instead of D?2A would make it somewhat more easy to digest?

Sometimes the executor description is in the text content of the type="text" parameter "uei-executor",

Code: Select all

<irp:parameter name="uei-executor">01F8[D;F]</irp:parameter>

sometimes it is in subelement of a type=xml" parameter, like

Code: Select all

    <irp:parameter name="uei-executor" type="xml"> 
            <rm:deployment executor="0162[D,Parm;F]"> 
                <rm:protocolName>DirecTV</rm:protocolName> 
                <rm:assignment target="Parm">1</rm:assignment> 
            </rm:deployment> 
        </irp:parameter> 

please be consistent. (Probably it would be best to use the latter form for all.)

[mathdon]

I am amenable to re-naming the document as other than an Appendix to the IRP Spec. I was also going to remark that there should be an Appendix to the IRPSpec documenting your additions to the notation. I would not presume to write it, though. I think that is for you to do. I make no copyright claim on the IRP Spec, which is still described as a draft, and am entirely happy instead for the main document to be edited to include your additions. (I can supply it as a .doc file rather than .pdf if desired). I will add a reference to your IrpTransmogrifier documentation in the next draft of my new document. My apologies, no slight intended. Please let me know of any other references and/or copyright statements that you want me to add in either documentation or the source code.

Ease of use is not the number of characters the user needs to type, rather it is the number of characters the author needs to explain to the user...

This notation is being used in exactly one document that almost certainly has just two human users, you and me. All other users are computers that have no difficulty with it. If in due course anyone else wishes to edit my executor descriptors or write additional ones, I have provided a detailed document on the notation.

Alternatively, these parts could have been packed into XML elements and attributes, avoiding the parenthesis-bracket-brace-semicolon... magic.

Possibly, but I have neither the time nor the desire to figure out how to do this. The examples you have provided in our earlier exchanges suggest this would be very verbose and, to me at least, more difficult to understand than my concise one-line notation which was easy to write for all the 100+ protocols. With XML I would probably never have started and certainly never have finished the task.

[mathdon]

NOTE: I wrote this message before our exchange in the two posts above. I hope its tone is not over-optimistic in view of that exchange.

I have added IrpProtocols_2019-10-11-rm2.xml to the files in the RMIR Development folder on SourceForge. The changes from the one in yesterday's RMIR v2.09 build 5 are as follows:

* I have added a section of "Additional notes on usage in RMIR" below your section of notes at the top of the file.
* I have updated its version suffix from rm1 to rm2.
* I have converted the uei-executor parameter to XML form for Anthem_relaxed, Blaupunkt_relaxed and F12_relaxed so they show in RMIR without the "_relaxed" and with a comment "Non-standard repeats".
* I have restored prefer-over Pioneer to NEC1 (which I had commented out).
* I have restored the frequency of Pioneer-Mix to 40k and added frequency-lower and frequency-upper entries as for Pioneer (previously I had frequency 40.3k and tolerance of 700, and I forgot to correct this yesterday when doing so for Pioneer).

I think that from my point of view, this is now stable - that is, subject only to normal bugfixes and the like.

I have also done a comparison with the version in IrpTransmogrifier 1.2.0 to look for unintended differences. The only one was in Pioneer-Mix, now fixed as described above. I believe a complete list of differences to be:

1. The new additional notes, the version suffix and the xmlns:rm entry.
2. All the uei-executor parameters.
3. My replacement of your DirecTV protocol with the six protocols DirecTV_Px for x=0-5.
4. G.I.4DTV prefer-over G.I.4DTVold, though I agree it is best to delete the latter protocol.
5. My addition of protocols OrtekMCE_relaxed, SonyDSP, SonyDSP_relaxed.
6. The addition of frequency-lower 39700 and frequency-upper 41000 entries to Pioneer-Mix
7. The addition of prefer-over NEC2 to Pioneer-Mix (which should have a predicate, as in your Pioneer entry, but I don't know what it should be).

I think that 6 and 7 are probably oversights when you updated Pioneer and I hope the rest are now non-controversial. So I hope we are getting to a position we both consider stable. But what is your intention about the documentation entries in IrpProtocols.xml? There are over 50 references to DecodeIR, which seem out of place.

[mathdon]

What was named Appendix 1 of the IRP Specification is now Executor Descriptor Notation and is in Help Files, though the link is the same. The Introduction has been split into two parts, the first of which is new. It describes IrpTransmogrifier, gives a link to it and credits it to you. It also gives a link to RMIR and explains the use of executor descriptors as a link between the two.

Otherwise it just has consequential changes to remove any mention of it being an Appendix to the IRP Spec. I appreciate you taking the time to suggest changes, but I hope we can agree to differ on them. Names can already be multi-character but, like the norm in IRP notation, I prefer the conciseness of single letters. Your suggestion on selector notation adds little to digestability in my opinion. The use of type="xml" for all uei-executor content would add approaching 500 lines to the document for no benefit other than a perceived consistency. You use the parenthesis-bracket-brace-semicolon... magic that is IRP notation so a second element that uses something similar does not seem a big issue to me.

In the RMIR Development folder, IrpProtocols_2019-10-11-rm2.xml has been replaced by IrpProtocols_2019-10-11-rm3.xml. The sole change is that it now references the Executor Descriptor Notation document and gives a link to it, rather than mentioning Appendix 1 in my notes on its use in RMIR.

In build 6 of RMIR v2.09, Help > About contains a new sentence "Incorporates IrpTransmogrifier by Bengt Martensson". As far as I am concerned, build 6 (not yet posted) is ready for general release to give it a wider airing. But as it incorporates IrpTransmogrifier, I would prefer to have your OK before doing so.

[mathdon]

I have just noticed the following note in the IrpTransmogrifier documentation:

Performance consideration, both time and space, were given secondary priorities. However, the decoding mechanism is intrinsically much slower that DecodeIR. A single decode can take several hundred milli-seconds.

Is this still valid? When I open the Learned Signals tab in RMIR with test file 1, for example, which has 159 learned signals, it opens with a display of decodes of all these signals in less time than I can measure, perhaps 200ms for the whole lot. I have been mightily impressed with its speed .

[Barf]

IrpTransmogrifier 1.2.1 has been released. This allows for several configuration files, where the later overwrites the earlier. There is also from API an IrpDatabase.patch(File) function, allowing the same thing to be made from the API. There is also a dump for writing existing data base (which could have been created from several files) to a file. Also Graham's changes has been merged. The protocol documentation has been updated.

Release notes:

Code: Select all

* Merge Graham Dixon's fixes https://www.hifi-remote.com/forums/viewtopic.php?t=101943&start=63
  in particular in <irp:parameter name="uei-executor">.
* Updated protocol documentation etc in IrpProtocols.xml. Resolves #146.
* Implemented IrpDatabase.patch(File) etc, allowing for patching IrpDatabases.
  From command line, allow several arguments to --config. Resolves #68.
* New option list --dump. Resolves #144.
* Rename IrpDatabase.toDocument to IrpDatabase.toXml
* Protocols: G.I.44DTVold removed; G.I.4DTVnoCheck -> G.I.4DTV_relaxed.
  Remove all usable="true". Reordering.

[Barf]

... I will add a reference to your IrpTransmogrifier documentation in the next draft of my new document. My apologies, no slight intended. Please let me know of any other references and/or copyright statements that you want me to add in either documentation or the source code.

Actually, I pointed out a logical flaw in your text (it does now work with (bare) expression not having the "==" operator, for example), like a gap in a mathematical proof; I was not asking for credit... But thank you anyhow.

Performance consideration, both time and space, were given secondary priorities. However, the decoding mechanism is intrinsically much slower that DecodeIR. A single decode can take several hundred milli-seconds.

Is this still valid? When I open the Learned Signals tab in RMIR with test file 1, for example, which has 159 learned signals, it opens with a display of decodes of all these signals in less time than I can measure, perhaps 200ms for the whole lot. I have been mightily impressed with its speed Very Happy .

Basically yes. Performance was never first priority, while still avoiding too slow constructs and without writing Windows Vista-programs. I have run the program through the Java profiler in Netbeans, and there are no obvious bottlenecks (but there are too many exceptions thrown by failed decode attempts). See also this issue. But you are right, speed is currently not an issue, and frankly speaking also I am slightly surprised.

[mathdon]

Thank you for merging my changes/additions, and for the patch option that I have yet to investigate. But you seem to have made some changes to the decode algorithm that have made things worse. With my test file 1, I now get a number of signals that show up in RMIR as XMPFinalFrameMeta. This isn't from the database merge, as I have tested with the same database with v1.2.0 and 1.2.1 of IrpTransmogrifier. Signal #9 now XMPFinalFrameMeta, was correctly MCE. Signals #92, 101, 103, 106 now XMPFinalFrameMeta, were correctly OrtekMCE (or OrtekMCE_relaxed). Signals #97, 98, 99, 104, 105, 109, 110, 132 were OrtekMCE_relaxed (showing as OrtekMCE Incomplete signal), now None. Do you know what is causing this?

[Barf]

Sigh....

Pretty much came together there. First I had goofed up the xml-properties parsing so the executor mapping was missing, or in quite mutilated form as a text property. Secondly, also the <irp:protocol usable="false"....> was broken, which showed up as the unexpected occurance of XMPFinalFrameMeta. In the lines (LearnedSignal.getDecodes()) you write

Code: Select all

     if ( sigDecodes == null || sigDecodes.size() == 0 )
          {
            int[] dur = Arrays.copyOfRange( ul.durations, ul.oneTime, ul.durations.length);            
            IrSignal irSignal2 = new IrSignal( dur, 0, ul.repeat, ul.frequency);
            sigDecodes = tmDecoder.decodeIrSignal( irSignal2, tmDecoderParams );
          }

which I do not understand the purpose of. It managed to construct an empty signal, and call the decoder on that. Due to the second bug, the usable="false" protocol XMPFinalFrameMeta matched...

Both problems are fixed in the continuous integration build, please try that.

Sorry, embarrassing...

[mathdon]

Thank you, all seems well with 1.2.2-SNAPSHOT. All my test files produce identical decodes with that and with 1.2.1.

In the lines (LearnedSignal.getDecodes()) you write

Code: Select all

     if ( sigDecodes == null || sigDecodes.size() == 0 )
          {
            int[] dur = Arrays.copyOfRange( ul.durations, ul.oneTime, ul.durations.length);            
            IrSignal irSignal2 = new IrSignal( dur, 0, ul.repeat, ul.frequency);
            sigDecodes = tmDecoder.decodeIrSignal( irSignal2, tmDecoderParams );
          }

which I do not understand the purpose of.

If a decode fails, this repeats it omitting the one-time durations. I thought it would be made redundant by setIgnoreLeadingGarbage( true ), but that is not the case. If I comment out that section of code, a number of signals fail to decode that do so with it present. Examples in test file 1 are #15, 24, 35, 66, 67.

If you post the snapshot as 1.2.2 release version, I will then issue RMIR v2.09 build 6 also as an official release.

Build 6 accesses IrpProtocols.xml directly from IrpTransmogrifier. Instead of the separate RMIR version of the file, there is now an optional rmProtocols.xml which is a patch file to the file in IrpTransmogrifier, included only when needed, so it will be absent in build 6. I have also added a menu item Help > IrpTransmogrifier Protocols which opens IrpProtocols.html.

[Barf]

IrpTransmogrifier 1.2.2 has been released. This is a bugfix release.

Release notes

Code: Select all

===============================================================================
2019-10-20: Version 1.2.2

* Bugfix: Fix implementation of IrpDatabase patchfile nuking by empty entries.
* Bugfix #148.
* Bugfix #147.
* IrpProtocols.xml: deleted unusable  protocols.
* New constructor Decoder.SimpleDecodesSet(DecodeTree), new function Decoder.DecodeTree.toList().

[mathdon]

As you will doubtless have seen, I have now officially released RMIR v2.09 build 6, which incorporates IrpTransmogrifier 1.2.2. Many thanks for all the help you have been with this.

BTW I think this should be an Announcement thread so that it does not get buried if it now goes quiet. You may also have noticed that I have at last discovered how to do my own tagging of RMIR releases, so no longer need to call on your kind assistance .

[Barf]

First shot at documentation for the patch files:

More than one protocol data base file ("IrpProtocols.xml") can be deployed. This gives a user or a program the possibility to strictly divide his/her/its own entries from the official ones. A "small" file modifying and/or adding a few protocols is often called a "patch file", although there is actually no difference between the first and subsequent files, and the semantics is identical. All files are required to be valid XML, and should be valid with respect to the given XML schema.

From the command line, the option --configfile (short form: -c) can be given several times, or several files can be given as argument, separated by commas (","). From the API, see the function IrpDatabase.patch(File) (and others).

The content of the patch file is basically merged into the data base, amending the information already there, with the exception that an empty entry deletes the original one. The exact rules are as follows:

If a patch file contains an empty protocol element, the protocol with the same name will be (if present) removed from the data base. Otherwise, its content is amended into the present one.

For the protocol properties (both the XML properties and the normal ones) similar rules apply: An empty property element removes that property from the protocol. A non-empty property in a protocol in the patch file is added to the end of the list that is the value of that property in the protocol (unless it is already present).

However, the "properties" irp and documentation are different, since there can only be one of those in a protocol. For these, an entry in the patch file overwrites the original entry.

For an example, see the files (in the sources) src/test/resources/IrpProtocols-test.xml and src/test/java/org/harctoolbox/irp/IrpDatabaseNGTest.java (function testPatchFile()).

[Barf]

... BTW I think this should be an Announcement thread

Rob, can you take care of this? Thanx in advance.