IrpTransmogrifier: new program/library for IRP protocols

[mathdon]

Barf, on DirecTV you wrote

(You probably mean the other way around, since matching the longer implies matching the shorter.) In principle do-able, but I doubt that it will work "in practice"; the only way to really determine the intended lead-out it to send the signal immediately followed by something else.

Yes, I did mean the other way round. But the really interesting thing is that IT WORKS , tested on real DirecTV learns. I set up six upgrades on one remote, one for each parm value 0-5, and learned the same signal from each on to another remote. I also created six protocols, DirecTV_P0 through DirecTV_P5, one for each parm value, and the six learned signals show up in RMIR as from the appropriate protocol. BTW It does not work in DecodeIR, which distinguishes on frequency but gives the parm value of the longer leadout for both shorter and longer ones.

This brings me to a query. I would prefer to report these not as six different protocols, but all as DirecTV with the parm value in the Misc field in RMIR. I think I could do that by using your new feature of including XML in a parameter element. I would use two XML entries within the uei-executor parameter, one giving a name (DirecTV) that overrides the protocol name in RMIR and the other that adds text (the parm value) to show in the Misc field in RMIR. But I don't know how to access these entries. Could you give me an example, please?

[Barf]

How 'bout this:

In IrpProtocols.xml

Code: Select all

<irp:protocol c-name="DirecTV_P3" name="DirecTV_P3">
        <irp:parameter name="prefer-over">DirecTV</irp:parameter>
        <irp:parameter name="uei-executor" type="xml">
            <rm:deployment executor="DirecTV" pid="0162">
                <rm:protocolName>**DirecTV**</rm:protocolName>
                <rm:assignment target="Parm">3</rm:assignment>
                <rm:assignment target="CmdParms">F</rm:assignment>
                <rm:assignment target="DevParms">100*D + Parm</rm:assignment>
            </rm:deployment>
        </irp:parameter>
        <irp:irp><![CDATA[{38k,600,msb}<1,-1|1,-2|2,-1|2,-2>([10][5],-2,D:4,F:8,C:4,1,-50){C=7*(F:2:6)+5*(F:2:4)+3*(F:2:2)+(F:2)}[D:0..15,F:0..255]]]></irp:irp>
    </irp:protocol>

Here "rm" is a (dummy) XML namespace, that needs to be declared in the top element (line 4, the URI has no real meaning, therefore the URL of this thread ):

Code: Select all

<irp:protocols xmlns="http://www.w3.org/1999/xhtml"
               xmlns:xi="http://www.w3.org/2001/XInclude"
               xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
               xmlns:rm="https://www.hifi-remote.com/forums/viewtopic.php?t=101943"
               version="2019-09-11"
               xsi:schemaLocation="http://www.harctoolbox.org/irp-protocols http://www.harctoolbox.org/schemas/irp-protocols.xsd"
               xmlns:irp="http://www.harctoolbox.org/irp-protocols">

and to transmogrify:

Code: Select all

package javaapplication8;

import java.util.*;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.harctoolbox.ircore.*;
import org.harctoolbox.irp.*;
import org.harctoolbox.irp.Decoder.Decode;
import org.harctoolbox.irp.Decoder.DecoderParameters;
import org.w3c.dom.*;

public class directv2 {

    public static void main(String[] args) {
        try {
            IrSignal irSignal = Pronto.parse("0000 006D 000A 000A 00E4 002E 002E 002E 0017 0017 0017 0017 002E 0017 0017 0017 002E 0017 002E 002E 0017 0017 0017 0474 0072 002E 002E 002E 0017 0017 0017 0017 002E 0017 0017 0017 002E 0017 002E 002E 0017 0017 0017 0474");
            DecoderParameters decoderParameters = new DecoderParameters();
            IrpDatabase irpDatabase = new IrpDatabase("../IrpTransmogrifier/src/main/resources/IrpProtocols.xml");
            ExecutorWrapperDatabase executorWrapperDatabase = new ExecutorWrapperDatabase(irpDatabase);
            Decoder decoder = new Decoder(irpDatabase);
            Map<String, Decode> sigDecodes = decoder.decodeIrSignal(irSignal, decoderParameters);
            for (Decode decode : sigDecodes.values()) {
                System.out.println("Orig: " + decode);
                NamedProtocol namedProtocol = decode.getNamedProtocol();
                String protocolName = namedProtocol.getName();
                Map<String, Long> names = decode.getMap();
                NameEngine nameEngine = new NameEngine(names);
                List<ExecutorWrapper> wrapperList = executorWrapperDatabase.get(protocolName);
                if (wrapperList == null)
                    continue;
                for (ExecutorWrapper ew : wrapperList) {
                    Decode newDecode = ew.fixDecode(protocolName, namedProtocol, nameEngine);
                    System.out.println("Fixed: " + newDecode);
                }
            }
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }

    private static class ExecutorWrapper {

        private static final String RM_NAMESPACE = "https://www.hifi-remote.com/forums/viewtopic.php?t=101943";

        static List<ExecutorWrapper> parseList(List<DocumentFragment> exec) {
            List<ExecutorWrapper> result = new ArrayList<>(16);
            exec.forEach((fragment) -> {
                result.add(new ExecutorWrapper(fragment));
            });
            return result;
        }

        private String protocolName;
        private final Map<String, Expression> assignments;

        ExecutorWrapper(DocumentFragment fragment) {
            assignments = new LinkedHashMap<>(4); // Important: keeps the order things are put in.
            protocolName = null;
            NodeList children = fragment.getChildNodes();

            // Finding the deployment element
            Element deployment = null;
            for (int i = 0; i < children.getLength(); i++) {
                Node node = children.item(i);
                if (node.getNodeType() != Node.ELEMENT_NODE)
                    continue;
                Element e = (Element) node;
                if (e.getLocalName().equals("deployment")) {
                    deployment = e;
                    break;
                }
            }
            if (deployment == null)
                return;

            // Getting the assignments
            NodeList assignmentNodes = deployment.getElementsByTagNameNS(RM_NAMESPACE, "assignment");
            for (int i = 0; i < assignmentNodes.getLength(); i++) {
                Element e = (Element) assignmentNodes.item(i);

                // Get and parse the Expression contained therein
                Expression exp = Expression.newExpression(e.getTextContent());

                // and stuff it into the map
                String paramName = e.getAttribute("target");
                assignments.put(paramName, exp);
            }

            NodeList nl = deployment.getElementsByTagNameNS(RM_NAMESPACE, "protocolName");
            if (nl.getLength() > 0)
                protocolName = nl.item(0).getTextContent();
        }

        void fixParameters(NameEngine nameEngine) {
            assignments.entrySet().forEach((assgnmnt) -> {
                try {
                    String paramName = assgnmnt.getKey();
                    long value = assgnmnt.getValue().toLong(nameEngine);
                    nameEngine.define(paramName, value);
                } catch (NameUnassignedException | InvalidNameException ex) {
                    Logger.getLogger(directv2.class.getName()).log(Level.SEVERE, null, ex);
                }
            });
        }

        Decode fixDecode(String protocolName, NamedProtocol namedProtocol, NameEngine nameEngine) throws InvalidNameException, UnsupportedRepeatException, NameUnassignedException, IrpInvalidArgumentException {
            if (this.protocolName != null)
                protocolName = this.protocolName;
            fixParameters(nameEngine);
            NamedProtocol np = new NamedProtocol(protocolName, namedProtocol.getIrp(), null); // inefficient...
            Decode fixedDecode = new Decode(np, nameEngine.toMap(), -1, -1, 0);
            return fixedDecode;
        }
    }

    private static class ExecutorWrapperDatabase {

        private final Map<String, List<ExecutorWrapper>> map;

        ExecutorWrapperDatabase(IrpDatabase irpDatabase) {
            map = new HashMap<>(16);
            irpDatabase.getKeys().forEach((protName) -> {
                List<DocumentFragment> exec = irpDatabase.getXmlProperties(protName, "uei-executor");
                if (exec != null) {
                    List<ExecutorWrapper> lies = ExecutorWrapper.parseList(exec);
                    map.put(protName, lies);
                }
            });
        }

        List<ExecutorWrapper> get(String name) {
            return map.get(name.toLowerCase(Locale.US));
        }
    }
}

[mathdon]

Wow . Many thanks . That will take a lot of digesting .

[mathdon]

Thanks again . I have got this working in RMIR, at least as proof of concept, though requiring more work before I can include it in even a development build.

[mathdon]

I have now uploaded build 3 of RMIR v2.09 to the RMIR Development folder. I have made three uses of the XML-in-parameter feature to illustrate different aspects. The six DirecTV real learns in test file 4 now all show as DirecTV, with the Parm value showing in the Misc field. If converted to an upgrade - you can only convert one at a time as an upgrade must have all with the same parm value - then the parm value is set correctly. The three G.I.4DTV manufactured learns in test file 5 now all show as G.I.4DTV with the unit value correctly showing in the range 0-7. Finally I have added four manufactured learns of Amino to test file 3, two at 37kHz and two at 56kHz. They all show as Amino (rather than two at Amino-56) with the frequency in the Misc field, and they set the frequency correctly in an upgrade.

Barf, as you have indicated that you will solve the G.I.4DTV issue in a different way later, mine is just a temporary fix. As I have said, I will track any changes you make to IrpProtocols.xml. But as I found it easier to do so, for my temporary fix I have reverted to my workaround. There is nothing sinister in this so please don't take it amiss.

May I add that this use of XML within the uei-executor parameter gives flexibility for RMIR that is beyond my wildest dreams .

[mathdon]

I am about to go away on holiday, so there will be nothing further from me for ten days or so, at least.

[Barf]

I am about to go away on holiday, so there will be nothing further from me for ten days or so, at least.

[Wrong thread? ] I wish you a great vacation! (I have heard that in Britain there are pretty many going on vacation the next few weeks/month )

I am happy for the progress made on RMIR. The postprocessing that was done recently appears to work fine. However, I am slightly less impressed by the (original) protocol -> executor conversion. (Actually, it is conceptually not that different from the post processing). Can you describe, slightly more self-contained, syntax and semantics of the new notation?

I also intend to take some time of from this endeavor: Other things to do (also within "IT"...). The predicates in prefer-overs (discussed earlier in this thead, Issue) is almost finished. The mentioned refactoring of "equation solving" (Issue) is approaching completion. It will not change anything of the API (at least that is the goal). In the meantime, you have to live with the imperfections of the G.I.4DTV decode (after all, it is one of over 100 protocol, and a work-around is available.)

[mathdon]

[Wrong thread?] I wish you a great vacation!

Right thread . I thought you were the only person who might post something while I was away and expect a reply from me.

Can you describe, slightly more self-contained, syntax and semantics of the new notation?

I had started on that before I went away. I am writing it as an appendix to the IRP Specification. Here is the abstract:

Code: Select all

The IRP notation as specified in the Second Draft document dated 4 February 2010 is a notation for describing IR protocols.  This Appendix specifies an extension that enables the notation to be used also for identifying, for an IR protocol, an executor capable of generating the IR signals of that protocol.  It does not specify the executors themselves but instead identifies a particular executor from a list of executors and specifies how the numerical values that the protocol transmits should be mapped to input parameters of the executor.

The notation has evolved significantly since my first thoughts on linking IrpTransmogrifier to protocols.ini and as I realised the complications that could arise. There is still an element of post-processing the output of IrpTransmogrifier but that is not mentioned in the document as I see it as an implementation issue. Indeed, there is no mention of IrpTransmogrifier or of protocols.ini either, as I have tried to avoid anything that is implementation-specific. The notation as documented goes beyond build 3 of RMIR v2.09 but should be fully implemented in the forthcoming build 4. Writing the documentation forced me to think things through in more detail than I did during its gradual evolution, bringing to light issues that I hope I have resolved in the final (?) version.

In the meantime, you have to live with the imperfections of the G.I.4DTV decode (after all, it is one of over 100 protocol, and a work-around is available.)

I no longer have any issues with the G.I.4DTV decode, or any other decode, as the ability to use XML within a uei-executor parameter enables me to work around any such issues and hide them from RMIR. But as I have said before, I will track any changes you make to the working of IrpTransmogrifier so if you wish to make changes to the API, I will track them in RMIR.

[mathdon]

Barf, I have now posted a first draft of Appendix 1 to the IRP Specification in the Diagnosis Area. This is my first attempt at documenting the notation I am using in IrpProtocols.xml. In due course I would intend it to join the IRP Specification document in the Help files. However, I fully expect you and I to be its sole readership .

I look forward to your comments in due course.

[mathdon]

I have now uploaded build 4 of RMIR v2.09 to the RMIR Development folder. I believe this to be fully compliant with the first draft of Appendix 1 to the IRP Specification that I posted yesterday. There are no major differences from build 3 but writing the Appendix made me realize some omissions and inconsistencies that needed to be resolved.

[mathdon]

I misinterpreted the description of the Sony DSP protocol in protocols.ini where it says

If your learned signal decode displays 3 repeats of Sony15 36,96, then 3 Sony8 entries followed by one more Sony15 entry, then you need to use this Sony DSP protocol.

It says nothing further about the parameters of the final Sony15 signal so I presumed it implied that these again were 36,96. I have now tested RMIR v2.09 build 4 against real Sony DSP signals from the S3C80 executor in protocols.ini and found this to be wrong. I have looked at the disassembly of the executor and found that if F is the 8-bit OBC of the Sony DSP signal then the final Sony15 signal has device code 22+(F<128) and OBC (F&0x7F)^17.

Although Sony15 has a 7-bit F and an 8-bit D, the Sony DSP executor creates the Sony15 part of the Sony DSP signal from an 8-bit F and 7-bit D. The complete IRP for Sony DSP is also most easily expressed in this form, as

Code: Select all

{40k,600}<1,-1|2,-1>((4,-1,96:8,18:7,^45m)3,(4,-1,195:8,^45m),(4,-1,81:8,^45m),(4,-1,F:8,^45m),
      (4,-1,(F^145):8,11:7,^45m))[F:0..255]

However, further testing showed that the UEI learning algorithm seems to have a problem with this signal. The 11:7 at the end has its lsb binary form as 1101000 but remotes fail to learn the final 0. So to make RMIR recognize Sony DSP learned signals, I have added a SonyDSP_relaxed protocol that has 11:6 and omits the leadout ^45m, which is irrelevant as the overall signal does not repeat. This gives the following two entries in the IrpProtocols.xml file for RMIR:

Code: Select all

    <irp:protocol c-name="SonyDSP" name="SonyDSP">
        <irp:parameter name="prefer-over">SonyDSP_relaxed</irp:parameter>
        <irp:parameter name="prefer-over">Sony15</irp:parameter>
        <irp:parameter name="uei-executor">01FF:JP1DSP[;F]</irp:parameter>
        <irp:irp><![CDATA[{40k,600}<1,-1|2,-1>((4,-1,96:8,18:7,^45m)3,(4,-1,195:8,^45m),(4,-1,81:8,^45m),(4,-1,F:8,^45m),
            (4,-1,(F^145):8,11:7,^45m))[F:0..255]]]></irp:irp> 
        <irp:documentation/>
    </irp:protocol>
    <irp:protocol c-name="SonyDSP_relaxed" name="SonyDSP_relaxed">
        <irp:parameter name="prefer-over">Sony15</irp:parameter>
        <irp:parameter name="decode-only">true</irp:parameter>
        <irp:parameter name="uei-executor" type="xml"> 
            <rm:deployment executor="01FF:JP1DSP[;F]"> 
                <rm:protocolName>SonyDSP</rm:protocolName> 
                <rm:commentItem>Final data bit missing</rm:commentItem> 
             </rm:deployment> 
        </irp:parameter> 
        <irp:irp><![CDATA[{40k,600}<1,-1|2,-1>((4,-1,96:8,18:7,^45m)3,(4,-1,195:8,^45m),(4,-1,81:8,^45m),(4,-1,F:8,^45m),
            (4,-1,(F^145):8,11:6))[F:0..255]]]></irp:irp> 
        <irp:documentation/>
    </irp:protocol>

I'm posting this both for information and to see if you think this is a reasonable approach.

[Barf]

[Graham, I will get back to you shortly.]

IrpTransmogrifier 1.2.0 has been released!

There has been a considerable refactorization of the decode code. frequency-lower and frequency-higher as above has been implemented, just as the prefer-over predicates.
The G.I.4DTV issue has been solved -- the equation solving code has been completely rewritten.

There are some minor incompatible changes in the API, in particular that Decode.decodeIrSignal now delivers a Decode.SimpleDecodesSet (which implements Iterable<Decode>). Also, the attribute c-name in IrpProtocols.xml is now depreciated.

Release notes:

Code: Select all

* New parameters in IrpProtocols.xml: frequency-low & frequency-high. #132.
* IrSignal.setFrequency replaced by IrSignal(IrSignal, double). #143.
* Reorganized the contributed directory.
* Fixed protocol Humax 4Phase. #142.
* Cleanup/efficiency fix in Protocol.toIrSequence(NameEngine, Pass).
* IrSequence.approximatelyEquals(IrSequence): much stricter default tolerances.
* Bugfix: Number.toString(int) now behaves properly also for Integers etc.
* Implement index reversal for large bitspecs. #141.
* Major refacoring of decoding, using Equation and BitwiseParameter. #127, #137.
* IrpProtocols.xml/irp:protocol[@name="NEC1-f16"]: set reject-repeatless to true.
* IrpProtocols.xml: rename protocol Ortek -> Ortek_NEClike.
* Remove attribute c-name in IrpProtocols.xml; add an explaining comment. #140.
* Refactored much of the Decoder stuff.
* Implemented predicates for prefer-overs. #134.
* Adjusted prefer-overs in IrpProtocols.xml to be compatible with the past.
* Improvements to list --check-sorted (previously --checksorted).
* Better user messages for no decodes.
* Improved logging, in particular for rendering.
* Move --override to common options.
* "New protocol" F12_relaxed.
* Moved irp from NamedProtocol to Protocol.
* Move ProtocolNotDecodableException and ProtocolNotRenderableException from NamedProtocol to Protocol.
* Encapsulate (hide) expansion in IrpDatabase. #139.
* Fix implementation of the short circuiting operators && and ||. #138.
* Fixed render --decode in the presence of defaults. #135.
* Extended API of IrpDatabase. #136.
* IrpDatabase now implements Iterable<NamedProtocol>. Cf. #136.
* Comment character of --input and --namedinput now settable using --commentStart. #133.
* Bugfix: render --decode misbehaved on protocol with definitions. #135

Keep transmogrifying!

[Barf]

...
However, further testing showed that the UEI learning algorithm seems to have a problem with this signal. The 11:7 at the end has its lsb binary form as 1101000 but remotes fail to learn the final 0. So to make RMIR recognize Sony DSP learned signals, I have added a SonyDSP_relaxed protocol that has 11:6 and omits the leadout ^45m, which is irrelevant as the overall signal does not repeat. This gives the following two entries in the IrpProtocols.xml file for RMIR:

Code: Select all

    <irp:protocol c-name="SonyDSP" name="SonyDSP">
        <irp:parameter name="prefer-over">SonyDSP_relaxed</irp:parameter>
        <irp:parameter name="prefer-over">Sony15</irp:parameter>
        <irp:parameter name="uei-executor">01FF:JP1DSP[;F]</irp:parameter>
        <irp:irp><![CDATA[{40k,600}<1,-1|2,-1>((4,-1,96:8,18:7,^45m)3,(4,-1,195:8,^45m),(4,-1,81:8,^45m),(4,-1,F:8,^45m),
            (4,-1,(F^145):8,11:7,^45m))[F:0..255]]]></irp:irp> 
        <irp:documentation/>
    </irp:protocol>
    <irp:protocol c-name="SonyDSP_relaxed" name="SonyDSP_relaxed">
        <irp:parameter name="prefer-over">Sony15</irp:parameter>
        <irp:parameter name="decode-only">true</irp:parameter>
        <irp:parameter name="uei-executor" type="xml"> 
            <rm:deployment executor="01FF:JP1DSP[;F]"> 
                <rm:protocolName>SonyDSP</rm:protocolName> 
                <rm:commentItem>Final data bit missing</rm:commentItem> 
             </rm:deployment> 
        </irp:parameter> 
        <irp:irp><![CDATA[{40k,600}<1,-1|2,-1>((4,-1,96:8,18:7,^45m)3,(4,-1,195:8,^45m),(4,-1,81:8,^45m),(4,-1,F:8,^45m),
            (4,-1,(F^145):8,11:6))[F:0..255]]]></irp:irp> 
        <irp:documentation/>
    </irp:protocol>

I'm posting this both for information and to see if you think this is a reasonable approach.

Ok, using relaxed protocol is the only somewhat clean way to handle almost-hits. Two comments:

* Please do not use "c-name" any more, it is deprecated (not needed, just uglifies the data.) (Yes, I did not say this previously...)
* Any reason for using an element rm:commentItem instead of irp:documentation? The latter is in some circumstances presented to the user, for example by the "Docu" button in IrScrutinizer. (Of course, you may also use XML-style comments <!-- --> ).

[mathdon]

I have now uploaded build 5 of RMIR v2.09 to the RMIR Development folder. This incorporates IrpTransmogrifier 1.2.0 and has the RMIR version of IrpProtocols.xml correspondingly updated, including the deletion of all c-names.

Any reason for using an element rm:commentItem instead of irp:documentation?

Yes, it is not intended as documentation. It is a brief comment that RMIR includes in the Misc column on the Learned Signals tab, similar to corresponding text that DecodeIR provides. If you look at the learned signals in the new Test file 6, you will see this comment showing for the Sony DSP signals.

Two comments for you on v1.2.0:
* G.I.4DTV needs prefer-over G.I.4DTVold.
* You still have NEC1 prefer-over Pioneer, is this is an oversight?

I am still confused about Pioneer versus NEC2. I have a constructed signal, learned signal 157 in test file 1, that gives a multiple decode of NEC2 and Pioneer. It shows frequency 39801, D=90, S=165, OBC=38. I can't remember why I constructed this, but the D=90 appears to show that it isn't Pioneer. I just offer this for consideration, as it isn't a real signal but the multiple decode is not what I would have expected.

[The Robman]

I am still confused about Pioneer versus NEC2. I have a constructed signal, learned signal 157 in test file 1, that gives a multiple decode of NEC2 and Pioneer. It shows frequency 39801, D=90, S=165, OBC=38. I can't remember why I constructed this, but the D=90 appears to show that it isn't Pioneer. I just offer this for consideration, as it isn't a real signal but the multiple decode is not what I would have expected.

If the signal is approximately 40kHz, I would always assume this to be a Pioneer signal over an NEC2 signal. NEC1 and NEC2 are both 38kHz.