PAR scope and direction

Discuss the generic proposals for SJTAG
Post Reply
User avatar
Ian McIntosh
SJTAG Chair
Posts: 418
Joined: Mon Nov 05, 2007 11:49 pm
Location: Leonardo, UK
Contact:

Re: PAR scope and direction

Post by Ian McIntosh » Tue Dec 12, 2017 9:44 am

Following on from the discussions in yesterday's meeting, I'd agree with the view that the topic of "system fault metrics" falls well outside the area of concern for this group. Further, you could debate whether that should address the goodness of the test itself or the system's inherent capability for test: The latter does not guarantee the former. It may be that one of our outcomes is to suggest to TTSC that there should be a new study group to explore whether there is a demand for standardised system fault metrics and if so to produce a PAR for it.

Ultimately, no single group can hope to address everything that is related to system test so we need to "cut our cloth" so that what we do undertake is a manageable task. Indeed, "test" may be a rather flexible term as our Use Cases can encompass much more.

I can see security aspects as pertinent to access, and something that directly impacts what can be achieved by SJTAG. For example, we might use Aim 'B' to write configuration data into a FPGA, but if the Anti-Tamper Bit on the FPGA is set we may not be able to do so. However there are so many different aspects and implementations of "security" that I don't believe we should venture into that area other than to recommend that designs incorporate a method within the access protocols to authenticate a legitimate test or maintenance activity in order to "unlock" the security. You can't mandate that, because some regulatory requirement may prohibit it, or as in my FPGA example, it is physically impossible to do.

I think that also touches on something that Brad mentioned but was struggling to put into words: Within a standard, we will have aspects that are mandatory and must be implemented, others that are recommended and some that are optional: The wording of a standard typically refers to these as "Rules", "Recommendations" and "Permissions" for each specification of the standard.
Ian McIntosh
Testability Lead
Leonardo MW Ltd.

User avatar
Bradford Van Treuren
SJTAG Chair Emeritus
Posts: 103
Joined: Fri Nov 16, 2007 2:06 pm
Location: NOKIA / USA

Re: PAR scope and direction

Post by Bradford Van Treuren » Mon Jan 29, 2018 5:22 pm

Here is the content of the Needs Brainstorming discussion by the SJTAG Study Group on 29 January 2018. Please post your further comments using this forum page so we can keep a record of the ideas passed around this week before next meeting. Thanks.

SJTAG Need:
  • SJTAG is intended to improve the ability to test, diagnose and provide prognostic health information about systems.
    • (Analyze from top down in decomposition is necessary to be able to know what has to be exposed. How someone implements it is less important if it is clearly documented and usable. Testablilty “flow down” may be outside of SJTAG scope : Testability Framework Requirements. Available Testability “flow up” is what is advertised from the bottom up: Availability of Testability Features.)
  • A standardized method is needed to coordinate
    • (coordinate - exposure of underlying test capabilities that might exist?) (everyone puts testability at their level and don’t usually plan for use at a higher level) (Documentation of what is available at each level is key.)
  • component
    • (component could relate to discretes and not what we want)
  • access topologies,
    • (Board level BIST is more than a component access topology.)
  • interface constraints, and other dependencies at the board and system level
    • (Should really focus on system and sub-systems, which includes boards.)
  • in order to be able to effectively leverage the existing and future component level standards. Thus, a new supervisory standard is required to define the coordination and dependencies of instruments as well as configuration, management, and application of vector based testing at the board and system levels.
    • (The higher up you go in the hierarchy, the more you morph into functional testing.) (Downloading code into modules and executing them is also part of this infrastructure that is needed.)
Last edited by Ian McIntosh on Thu Feb 01, 2018 6:47 pm, edited 1 time in total.
Reason: Fixed spurious bullet at top of list
Bradford Van Treuren
Distinguished Member of Technical Staff
NOKIA MN

User avatar
Heiko Ehrenberg
SJTAG Vice Chair
Posts: 45
Joined: Wed Nov 21, 2007 3:15 pm
Location: GOEPEL Electronics - Austin, TX
Contact:

Re: PAR scope and direction

Post by Heiko Ehrenberg » Tue Jan 30, 2018 9:16 pm

SJTAG is intended to improve the ability to test, diagnose and provide prognostic health information about systems.
What about the "need" (?) to program / reconfigure modules / components in a system? Do we want to mention that in this list? Or do we want to focus on test and let programming/configuration follow naturally, utilizing the same methods put in place by SJTAG for the primary purpose of test (just like happened with JTAG)?
- Heiko

User avatar
Ian McIntosh
SJTAG Chair
Posts: 418
Joined: Mon Nov 05, 2007 11:49 pm
Location: Leonardo, UK
Contact:

Re: PAR scope and direction

Post by Ian McIntosh » Thu Feb 01, 2018 7:03 pm

The draft we started from was written to be Use Case independent because we recognised from the outset that SJTAG could be used for more than test. I guess that when 1149.1 was originally drafted, "test" was the only thing under consideration, and the other uses saw an opportunity to exploit what it provided.

We may need to consider whether talking in terms of "test" (rather than being abstract in our terminology) makes the standard more accessible; At the same time, I'm very aware that there's a large industry contingent for whom JTAG is something that you only use to program devices.

That maybe presents another conceptual question: Are we trying to convey the idea that "SJTAG = JTAG at a system level" or that "SJTAG = a system test scheme that happens to include JTAG".

To some extent we're caught by the long-standing perception that "JTAG means IEEE 1149.1" when it was actually meant to have a broader reach. At some point, might we have to set aside sematic correctness and just accept common usage?
Ian McIntosh
Testability Lead
Leonardo MW Ltd.

User avatar
Bradford Van Treuren
SJTAG Chair Emeritus
Posts: 103
Joined: Fri Nov 16, 2007 2:06 pm
Location: NOKIA / USA

Re: PAR scope and direction

Post by Bradford Van Treuren » Thu Feb 01, 2018 8:42 pm

Heiko's response is why I stated the sentence is good, but needs some tweaking. The points Heiko made are important. What we need is something in the middle of Louis' sentence and the original sentence. Both sentences represent the extreme end cases. The difficulty here is Louis' sentence is talking about use case, while the original sentence talked about functionality of the interface. They are not the same domain, but related. This gets back to my example like Grady Booche describes for object oriented modeling and the various perspectives you have to analyze a business domain with. It is a multi-faceted blob (more sides than a cube). You can look at the domain many ways. Some perspectives we have considered are:
  • By Structure (Topology)
    • Flat
    • Hierarchical
    • Retargeted
    • Transformed
  • By Behavior (Function)
    • By Stimulus/Response Behavior (API)
    • By State Change (Focus only on Target Instrument(s) as a Model of system)
    • By Hardware Signal protocol
    • By Software Data protocol
  • By Provisions
    • By Advertised/Exposed health directives
    • By diagnostic categories
    • By test categories (infrastructure, interconnect, BIST, etc.)
    • By test types (pure vector based, programming, monitoring, configuring, etc.)
  • By Decomposition
    • From Bottom Up Discovery
    • From Top Down Mining
  • By Protocol
    • Data Protocol
    • Access Protocol
    • Control Protocol
  • By Use Case
    • See SJTAG Universe Venn Diagram for examples
Each of these perspectives are important and all are part of what we call SJTAG in the purest sense. However, SJTAG, as a standard, needs to be the enabler for these perspectives to be realized within a system. That is where the "improve the ability to" comes into play. It is not just limited to a single list, but instead is encompassing a broad set of perspectives because no one person is going to use it exactly the same way as someone else. This is the very true case for 1149.1 and will be for SJTAG as well. The key with 1149.1 is it was clean and very clear on what could be done with it. It was an interface to registers inside of a device; one TDR happened to be able to control the pins of the device for structural test. The founders of 1149.1 built-in specific required elements (the required registers), but realized these were only examples of what users could really use with some innovation. We need to make sure SJTAG has that same kind of specificity as well as generalizations to apply the same standard for other uses outside of what we envision it for.
Bradford Van Treuren
Distinguished Member of Technical Staff
NOKIA MN

User avatar
Bradford Van Treuren
SJTAG Chair Emeritus
Posts: 103
Joined: Fri Nov 16, 2007 2:06 pm
Location: NOKIA / USA

Re: PAR scope and direction

Post by Bradford Van Treuren » Mon Feb 05, 2018 4:59 pm

Here are the notes from the Feb 5, 2018 meeting:
SJTAG Need:
  • More implementation than things in the PAR (e.g., First Red block) Need more than what is just listed in first bullet. Need as part of design so it can be part of the end game (DFT, DFM, etc.).
  • Still need some reference to “What is a good test?”
  • “What constitutes a Test Coverage?”
  • Everyone is working on the same thing (going in same direction) using standard as a template.
  • Not in formal Need, but part of standard.
  • May be some scenarios where user does not care about level of test coverage, but more just about if topology is operational.
  • Some cases where test does not even come into the occasion (e.g., programming use case).
  • These are still requirements for what the test infrastructure has to accomplish.
  • Needs of a product may be variable, but may not be of interest to some people in the product life cycle.
  • Probably don’t need to dig into such details for the Need statement.
  • Need to improve capability of what can be done as part of test access. But how can you validate such improvement? Can’t deal with entire product development life cycle.
  • The way we use SJTAG is going to be different whether we use it for DVT, Integration Test, Mfg. Test, etc. Ultimately, we need to know what are all the things we need to be handling as SJTAG. That is what should go in the Needs section. The Needs section is the only section where this kind of discussion can go. Do we need to flesh all this out in the PAR. Probably not. This is what the “word smithing” is all about. This is what the initiative group originally did with the perspectives from that team. There are additional perspectives now in this team. We are not driving a capability into hardware. Instead, we are attempting to leverage what is available in these devices and assemblies to perform enhanced capabilities.
  • Need to know what is available to work with (Terry’s top down idea). Some features in a device may not lend themselves to using at a higher level. It causes more problems at the assembly level then help. 1149.1-2013 tried to identify the Test Mode to help caution for cases of catastrophic state change of a board requiring a total reboot after a test. None of these features are visible with the latest interfaces on market.
Last edited by Bradford Van Treuren on Mon Feb 19, 2018 2:20 pm, edited 1 time in total.
Bradford Van Treuren
Distinguished Member of Technical Staff
NOKIA MN

User avatar
Heiko Ehrenberg
SJTAG Vice Chair
Posts: 45
Joined: Wed Nov 21, 2007 3:15 pm
Location: GOEPEL Electronics - Austin, TX
Contact:

Re: PAR scope and direction

Post by Heiko Ehrenberg » Mon Feb 05, 2018 7:20 pm

Perhaps it would help address Louis' concerns if we include a sentence in the Needs statement that says that SJTAG recognizes the need for some form of standardized measurement of test coverage and quality of test, but this standardization effort (this PAR) does not attempt to address that need?
- Heiko

User avatar
Ian McIntosh
SJTAG Chair
Posts: 418
Joined: Mon Nov 05, 2007 11:49 pm
Location: Leonardo, UK
Contact:

Re: PAR scope and direction

Post by Ian McIntosh » Mon Feb 05, 2018 9:21 pm

That sounds like a fair suggestion, Heiko.

A thought occurred to me this evening: We've spoken about SJTAG as providing improved test coverage but the thought I had was that need not be true for SJTAG to still be successful. A tool need not only allow you to do some task you couldn't otherwise do at all, but may allow you to do something you could already do, but with much less work.

I think that was one of our early motivators: Not so much that SJTAG makes tests possible, but that it makes it easier for test generation tools to understand what is possible.
Ian McIntosh
Testability Lead
Leonardo MW Ltd.

User avatar
LYUngar1
SJTAG Member
Posts: 9
Joined: Thu Nov 30, 2017 9:45 pm

Re: PAR scope and direction

Post by LYUngar1 » Tue Feb 06, 2018 6:39 pm

I would be OK with Heiko's suggestion. I think we could say it something like this:

"There is a need to ease test development and to improve test coverage measured by various means including, but not limited to, design verification tests (DVT), standardized failure mode effects [criticality] analyses, FMEA/FMECA, and test requirements documents (TRDs). While this PAR does not address such metrics, it assumes that an acceptable test coverage metric exists that SJTAG can improve. SJTAG can accommodate this need by improving the ability to test, diagnose and provide prognostic health information about systems...

User avatar
Bradford Van Treuren
SJTAG Chair Emeritus
Posts: 103
Joined: Fri Nov 16, 2007 2:06 pm
Location: NOKIA / USA

Re: PAR scope and direction

Post by Bradford Van Treuren » Tue Feb 06, 2018 9:55 pm

...may allow you to do something you could already do, but with much less work.
Ian is quite right. I have been doing ad hoc embedded boundary-scan testing since 1990 based on 1149.1 standard (actually started before that). The key to Ian's point is making it easier to do what I am already doing and do it with commercial tools that I don't have to support. That is embodied in his second comment.
I think that was one of our early motivators: Not so much that SJTAG makes tests possible, but that it makes it easier for test generation tools to understand what is possible.
The last point is I am leveraging the existing infrastructure to then perform other non-test tasks because it comes for free. I had to develop my own architecture and internal standard to make it useful internally. Still gaps in capability, but satisfies most current needs, albeit at possibly higher cost than could be accomplished with vendor tooling and a more standard, adaptable interface.
Bradford Van Treuren
Distinguished Member of Technical Staff
NOKIA MN

User avatar
Ian McIntosh
SJTAG Chair
Posts: 418
Joined: Mon Nov 05, 2007 11:49 pm
Location: Leonardo, UK
Contact:

Re: PAR scope and direction

Post by Ian McIntosh » Mon Feb 12, 2018 3:17 pm

Heiko Ehrenberg wrote:
Mon Feb 05, 2018 7:20 pm
... SJTAG recognizes the need for some form of standardized measurement of test coverage and quality of test, but this standardization effort (this PAR) does not attempt to address that need?
LYUngar1 wrote:
Tue Feb 06, 2018 6:39 pm
There is a need to ease test development and to improve test coverage measured by various means including, but not limited to, design verification tests (DVT), standardized failure mode effects [criticality] analyses, FMEA/FMECA, and test requirements documents (TRDs). While this PAR does not address such metrics, it assumes that an acceptable test coverage metric exists that SJTAG can improve. SJTAG can accommodate this need by improving the ability to test, diagnose and provide prognostic health information about systems...
I'm more comfortable with the more compact form of Heiko's suggestion (although it maybe wasn't actually presented as proposed form of words) - it seems a better fit to the concise style of language expected in a PAR. That said, Louis' suggestion includes elements that fit into the more general aims of SJTAG ("ease test development", "improving the ability to test, diagnose ...") that do need to be included in the PAR (somewhere). IMO, I think I'd shy away from appearing to recommend particular methods of coverage measurement by mentioning them.
Ian McIntosh
Testability Lead
Leonardo MW Ltd.

User avatar
Bradford Van Treuren
SJTAG Chair Emeritus
Posts: 103
Joined: Fri Nov 16, 2007 2:06 pm
Location: NOKIA / USA

Re: PAR scope and direction

Post by Bradford Van Treuren » Mon Feb 12, 2018 5:04 pm

Notes from February 12, 2018 meeting:

SJTAG Need:
  • A standardized method is needed to coordinate component access topologies, interface constraints, and other dependencies at the board and system level in order to be able to effectively leverage the existing and future component level standards. Thus, a new supervisory standard is required to define the coordination and dependencies of instruments as well as configuration, management, and application of vector based testing at the board and system levels.
  • Ideology vs. interface
  • Specific interfaces are not really broad enough for the Needs statement.
  • More widespread parts are the more need for a standard to bridge them together (aiding to PnP of COTS)
  • What is there and what you can make use of
  • Leverage lower level stds perspective is bottom up
  • Do we need 2 stds: Top-down defining sub-system “interfaces”, interoperability with low level stds
  • Req’s flow naturally goes top-down. Component BOM drives what is testable in the design (bottom-up). A hybrid approach is needed.
  • Enabler of the collective capabilities for the system test. Bottom-up needs to know what is going to be compatible to SJTAG.
  • SJTAG Compliance Levels: Range from general health to specific access/data interfaces to exiting standards
Bradford Van Treuren
Distinguished Member of Technical Staff
NOKIA MN

JCSTEW123
SJTAG Member
Posts: 3
Joined: Wed Sep 20, 2017 2:30 pm

Re: PAR scope and direction

Post by JCSTEW123 » Mon Feb 12, 2018 7:13 pm

Good meeting today. Looks like I should be on the call next week as well. Just engaging on the forum and making sure I have my email setup as well for notifications. :D

User avatar
LYUngar1
SJTAG Member
Posts: 9
Joined: Thu Nov 30, 2017 9:45 pm

Re: PAR scope and direction

Post by LYUngar1 » Tue Feb 13, 2018 10:35 pm

I have been accommodating to all types of suggestions to "tone down" the need for test coverage metrics, but it appears that it is still not being included - even in the Need section of the PAR. If better test coverage of systems is not part of the Need, then I submit there is no need at all for testability, thus no need for SJTAG. Why have testability if it doesn't improve test coverage? If testability and SJTAG do improve test coverage, why hesitate to mention that? In my previous post here I highlighted 3 different test coverage metrics as examples of how one MAY gauge test coverage. One example had to do with design verification tests, the second with failure modes or defects, and the third with diagnosis. Though I did not refer to any particular standard, these test methodologies are standardized. Since the three test coverages are different and since I believe all three would benefit from SJTAG testability, I decided all three should be included, even if it made the sentence wordy.

I feel very strongly that for SJTAG to be accepted, it needs to stand on solid footing and without improved test coverage being that foundation, SJTAG serves nobody. It becomes a "wouldn't it be nice" feature but would ultimately not solve the problems I think NEED to be solved.

User avatar
Bradford Van Treuren
SJTAG Chair Emeritus
Posts: 103
Joined: Fri Nov 16, 2007 2:06 pm
Location: NOKIA / USA

Re: PAR scope and direction

Post by Bradford Van Treuren » Wed Feb 14, 2018 12:04 am

I would have to courteously disagree with Louis. Test Coverage is not the only reason for the existence of SJTAG. There are many factors, too many to enumerate in a PAR.
  • Example 1: Standardized interface/mechanism for embedded instrument access by product application code.
    • IEEE 1687 makes access to registers in devices far to complex for application software to be efficiently written to access such register with embedded software. Thus, no one is wanting to use 1687 in their designs unless they have a lot of in-house people to support it and a budget to support ad hoc development of software to talk to such device instruments.
    • 1687 only goes to the device edge for automation of code to support such instrument register access. Systems need access from the board and higher levels if these registers are necessary for normal product function.
    • 1149.1-2013 suffers from the same problem in that it stops at the device edge.
    • SJTAG has a huge potential for filling the gap for code automation and vendor tooling in a standardized way to reduce the cost of supporting such complex architectures. Without this, none of these device standards will be usable for functional access and application for a product; only for test purposes. Thus, functional application registers will not be able to be implemented by standards like 1687 and 1149.1-2013.
    • Many registers in complex devices are accessed upon initialization to configure the device to a particular structure, behavior, or functionality. These are typically, access-once kind of registers. Even to talk to these require a device driver in the embedded code supporting the particular interface. I2C and SPI are straight forward. 1687 and 1149.1-2013 are quite complex and require special skill with no OTS driver available from the device vendor.
    • I am seeing more and more where device vendors are providing code libraries to support their devices and enforcing a specific hardware design and specific hardware hosts they will support. Thus, it hides from their customer the details of the interface using a simple library API that the embedded code links to. If a vendor has to write that library by hand, it is quite costly and they will probably miss their delivery window required by the customer. Having a standard providing support to simplify this code development is HUGE.
  • Example 2: Simplification of DVT testing prior to Software available on product
    • New products always struggle with testing of a new design because there is no access to controlling registers in the hardware.
    • Even the current emulation tools provide some level of peek and poke to memory mapped registers that is available by various vendors.
    • The problem comes in accessing non-memory-mapped registers.
      • How do I access I2C based registers? Especially those hanging off an FPGA PL side pins.
      • The same goes for SPI and other interfaces.
      • Adding an I2C Master instrument to the FPGA that is accessible from JTAG is part of the solution and automation of such instrumentation from the BSCAN tool vendors is certainly improving on this. BUT, there is nothing that defines how to connect that I2C Master to the target instruments for tooling to use in automating the I2C messages required to control such instruments. There is also a need to transform the I2C scoped messages into I2C Master directives over the JTAG interface to perform such I2C operations. That can all be automated to provide the code required if something like SJTAG is able to describe and manage such interfaces.
      • You can ask the BSCAN tool vendors how many companies use these downloaded interface masters for DVT. Not many because it is too costly to support and the BSCAN vendors have to charge so much for the service to recoup the development and support costs for each customer's application. That is because everything has to be done by hand.
      • So most products end up waiting for the software team to implement the basic OS and some rudimentary drivers to exercise important device registers for DVT. SJTAG could help me get my product to market faster than I do today. That is HUGE.
    • Now let's consider if a board does not boot in the system. There is nothing responding from system level diagnostic interface. You can be fortunate if it is an FRU and test it later, but what if many boards fail to boot in a system. How do you detect the root cause. Having these DVT tools available to acquire a snapshot of the state of each board, in-system, is HUGE advantage to understanding the health of the system.
    • This is not driven by test coverage, but by diagnostic granularity that is only available if such an advanced interface is available.
    • That interface needs to be something standardized and easily supported by tooling.
    • I have also analyzes the case where knowing the internal state of a board is actually beneficial for the service performed by the board. It may reveal the board is operational and the board management controller (BMC) fails to connect to the system services. The customer can still limp along without degradation of service while a spare is being dispatch for replacement. That can also be a HUGE cost savings.
So there are many other factors, only a couple named, that contribute to the Need for SJTAG. The point being made in the meetings is there are too many to enumerate in the space provided by a PAR format. I could argue my needs are more important than your needs, but that is not what this is about. We need to get the PAR done with just enough wording to make it scoped properly. As I said, we seem to be needing multiple PARs from this team.
Bradford Van Treuren
Distinguished Member of Technical Staff
NOKIA MN

Post Reply