As part of the Clean Software Alliance’s mission of championing sustainable, consumer-friendly practices across the software distribution ecosystem, we focus our resources in service of our constituents. This includes, when appropriate, conducting independent reviews of service providers and programs in the space. Earlier this year, after receiving numerous complaints and inquiries, the CSA embarked on a review of AppEsteem Corporation’s “deceptor” program and associated practices. Our findings may be found in the Report linked-to below.
Prior to its publication, the CSA offered to provide this Report in full to AppEsteem management, and to then sit down with AppEsteem management in person, to provide an opportunity to respond, to address any inaccuracies, and to take steps to ensure AppEsteem was employing best practices in its initiatives. AppEsteem unfortunately declined to engage with us. Nevertheless, the CSA still provided AppEsteem with an advance copy of the Report, so they could begin to digest its findings.
The CSA stands by our findings, and will continue to provide best-in-class resources, insight and guidance to our valued supporters, as well as the industry at large.
Scroll down to read the Report, or for a downloadable version, please click here.
* * *
A Review of AppEsteem Programs (May 2018)
Acting in the interest of its constituents within the distributed software ecosystem, the Clean Software Alliance (CSA, we or our) conducted a review of AppEsteem Corporation’s (AppEsteem) “deceptor” program. This review was triggered by a groundswell of complaints and expressions of concern received by the CSA from industry members regarding this program. The formal review commenced in earnest in February 2018, and over the ensuing ~8 weeks the CSA held in person meetings, conducted telephonic interviews, considered anonymous and aggregated submissions, reviewed AppEsteem’s own materials, and independently ran a comparison of AppEsteem certified and so-called “deceptor” applications.
Our findings are set forth below in this Report. Within our findings, two themes were particularly prevalent, namely, that (a) with virtual unanimity, each and every participant in our review expressed concerns with the deceptor program; and (b) at the same time, many respondents (AV respondents in particular) found potential value in establishing consensus-built, objective “minbar” criteria to bring further standardization and clarity of requirements to the ecosystem. Accordingly, while aspects of the program appear to show merit and should be further explored in cooperation with industry, the program is perceived by industry as being significantly flawed in its current design and implementation.
While our findings are encouraging, that encouragement is tempered by a sense of urgency surrounding the need for reforms to the program. Simply put, when dealing with a program with such far-reaching ambitions, the perception of industry that the program is seriously flawed in its design and its implementation requires immediate, substantive changes to remedy.
Separately, but importantly, we were also encouraged to find a small but growing number of industry members providing positive feedback regarding AppEsteem’s certification program generally, and its outsourced compliance services in particular. These positive sentiments are particularly encouraging to the CSA as they relate to the very services AppEsteem was originally formed to provide.
This Report concludes with a number of proposed steps, which we are hopeful AppEsteem’s leadership will duly consider. We are also hopeful that AppEsteem will see its way back to collaborating with industry through the CSA, and welcome the reopening of discussions regarding a closer relationship between AppEsteem and the CSA.
Per its earliest presentations to industry, AppEsteem was initially formed to provide certification services to developers and distributors of software products. These services were, at the outset, generally in alignment with the CSA’s goal to bring clean, sustainable practices to the software distribution ecosystem, and the CSA was supportive of many of AppEsteem’s early initiatives.
In early 2017 AppEsteem’s business model evolved. Rather than focusing on growing its certification and compliance-centric services organically, AppEsteem started rolling out its “deceptor” program. From the outset, the program elicited statements of concern from industry members, with many expressing these concerns directly to AppEsteem. Despite these concerns, AppEsteem pursued and expanded the program, and it has since become a focal point of AppEsteem’s presentations to industry, and its business development and PR efforts.
By the fourth quarter of 2017 and through the initial weeks of 2018, the CSA was hearing increased “chatter” regarding the deceptor program, with the breadth of volume of complaints and expressions of concern increasing. While the particulars of these communications were varied, virtually all focused on one or both of the following themes:
- The deceptor program lacks integrity in its design and implementation. Specifically, the program appears to be applied subjectively and with a double standard. This means that certain products one would expect to be on the deceptor list (assuming an even-handed application of the criteria) are not, in fact, on the list; and certain other products will be placed on the list and thereby suffer potentially undue and disproportionate punishment for unintentional or non-deceptive behavior.
- The deceptor program is an overly aggressive business development tactic. Specifically, AppEsteem aggressively deploys the program, to pressuring (a) monetization companies into associating with (and ultimately paying) AppEsteem; and (b) AVs to adopt the deceptor criteria as an industry standard, thereby amplifying AppEsteem’s reach and creating a vicious cycle.
The complaints and concerns reached a crescendo during and immediately following the AppEsteem meet-up that took place in Las Vegas in January 2018 at Affiliate Summit West. It was during this meet-up that concerns which previously had been expressed only in private conversations were suddenly thrust into the spotlight, specifically including the issues of a lack of trust and the application of double standards.
Following the Las Vegas meet-up, the CSA received an anonymous email that included a PDF showing alleged “deceptor”-level violations by multiple AppEsteem AV partners. Without taking a position on the contents or allegations, the CSA promptly shared the email and PDF with AppEsteem and its security partners for their consideration and response. A copy of the PDF, annotated by AppEsteem to reflect its response, and the cover email are available upon request, and may additionally be made available on the CSA’s website, www.cleansoftware.net.
That same week, the topic made its way into the next CSA Advisory Board call, triggering a spirited discussion among the participants (which, at the time, included Dennis Batchelder, President of AppEsteem). The Advisory Board call concluded in agreement that the CSA would conduct a review of the deceptor program and share its findings with AppEsteem. This plan was then shared with AppEsteem’s security partners and steering committee.
The CSA then mobilized its resources and embarked on its review, the proceeds of which are contained in this Report.
The CSA took a multi-faceted approach to its review of the deceptor program, deploying its resources in numerous ways. This included the following actions:
- Ensuring that the fact that the review was taking place was known by AppEsteem;
- Engaging in outreach to industry members to solicit input and feedback;
- Auditing the software included on the deceptor list and comparing against certified software and the software of others who support/work with AppEsteem;
- Reviewing AppEsteem-produced materials;
- Reviewing online and submitted materials;
- Convening a diverse group of attendees for two days of in person meetings. This group was comprised of AVs and monetization companies, and included AppEsteem clients, non-clients and security partners, and companies with products actively or previously included on the deceptor list.
- Conducting telephone interviews with industry members.
At all times, the CSA made clear its desire to hear all feedback – good, bad or indifferent – about the deceptor program and/or AppEsteem generally. To ensure appropriate levels of comfort and openness, the multi-participant meeting was held under Chatham House Rules, and anonymity was assured for all conversations, telephonic or in person. Efforts were made to ensure as balanced participation as possible during the entirety of the process.
The CSA received direct and indirect input from upwards of 50 industry members during the course of its review, including via in person meetings, telephone interviews, aggregated responses from service providers and anonymous submissions.
- 50+ total respondents
- ~10 AV/Security companies commanding over 63% of global Windows anti-malware market share (ex-China).
- 30+ non-AV software developers/distributors, including –
- ~10 companies who are or have been AppEsteem clients.
- ~10 companies who have or have had at least one product designated a “deceptor.”
- At least one representative from each of the following company types: browsers; AV testing organizations; download portals; trade associations; consulting firms.
Findings & Conclusions
As noted above, our findings coalesced around the macro issues of (a) flaws in the deceptor program’s design and implementation, and (b) abuse of the program as an overly aggressive business development tactic. That said, specific concerns were at times more granular, and at times broader. Indeed, what we found were concerns touching upon most aspects of the program. A common refrain was that AppEsteem was acting as judge, jury and executioner. Some felt it was even more problematic than that, with AppEsteem additionally acting as the legislature and law enforcement, as well.
What follows are our specific findings regarding the deceptor program and associated practices based on the inputs received:
- There is an apparent lack of industry involvement or consensus on deceptor criteria, and the criteria themselves suffer scope “creep” and subjectivity.
In the words of our respondents:
- “more deceptor criteria [being introduced] without input”
- “things have gotten subjective”
- “concerns that there is no approval process or communication to make determinations. No critical mass achieved to make something a deceptor criteria.”
- “the way ACRs are created and moved into deceptor status is problematic.”
- “AE says [certain products] are malicious, but [INTERVIEWEE] doesn’t see it that way; that is why there is a PUP program.”
- “[Deceptor pogram] should be based on industry rules, not AppEsteem rules.”
- “Implementation and execution are problems. Lack of consistency and transparency.”
While we note that AppEsteem holds calls with various groups that it works with, our findings show that only a limited audience is apprised of changes to the deceptor criteria; that this audience is not necessarily representative of the industry at large; the changes themselves are often presented as being final, rather than true (open) proposals; and the opportunity to provide feedback is often on an unreasonably tight timeframe.
- AppEsteem is deploying questionable targeting and enforcement tactics for its deceptor program.
Multiple respondents expressed their belief that AppEsteem specifically targets companies or classes of software in a manner designed to generate business or punish those unwilling to work with AppEsteem. At least one respondent, who was being pitched by AppEsteem to become a client, reported being asked which of their competitors they would “like to see on the deceptor list.” Other respondents reported conversations during which AppEsteem identified specific companies it was considering targeting as part of the deceptor program without any sound basis or jusitification.
- AppEsteem is engaging in aggressive marketing tactics, including leveraging the program to pressure companies into working with AppEsteem –
- pressuring monetization companies to sign up for AppEsteem services;
- pressuring AV industry members to adopt the deceptor construct as an industry standard;
In the words of our respondents:
- “Deceptor program is a scare tactic in itself.”
- “[Deceptor program is] a tactic to convert as many people as possible into a paid program.”
- “[Deceptor program is] a huge tool for business development [for AppEsteem]”
- “[Deceptor program] is a horrible way of generating customers.”
- “No question AE is using the [deceptor] program for business development.”
In fact, a clear majority of respondents believed the program is being run by AppEsteem as a business development tool, with some calling the program the “ultimate scareware tactic” to generate customers. This was precisely the view of one industry member who took to scambook.com to air his complaint. And another who forwarded an anonymous email to the CSA. A copy of these complaints are included as Appendix 1 and Appendix 2 to this Report.
Some other respondents were concerned that AppEsteem might be overstating its capabilities or potentially misleading prospective customers in its own marketing messages, in an effort to secure paying customers. This concern was most recently raised due to statements made by AppEsteem employees suggesting a deep, trusted relationship between AppEsteem and Microsoft. In the words of an AppEsteem employee:
“…As far as how Microsoft’s announcement will impact AppEsteem, we interpret the MS changes as largely aligning with AppEsteem’s approach and requirements. We’re in constant touch with them, and they trust us.
It’s certainly possible that they will decide to be even stricter than we are. The good news is, given our relationship with them (we meet with them frequently), I think we’ll find out immediately and be able to work with our Premium Customers to find out exactly what needs to be changed. That said, if you are on the Free program, if they decide to block any of your certified apps for any reason, you’ll just need to contact those folks on your own.
We are encouraging App Developers to get ahead of this Microsoft change by getting Certified as soon as possible…” (emphasis added)
The above was part of an email message sent to a prospective AppEsteem client. A copy of the email is included as Appendix 3.
When the CSA reached out to Microsoft for a clarification of its relationship with AppEsteem, the official response was as follows:
Microsoft does not have any special relationships with any third party data suppliers. Our detections are based on our own independent research and are within our sole discretion. Our research may include data and information from third party sources; however, our policy decisions and corresponding enforcement are independent, and third parties do not have the authority or permission to ‘white list’ any programs on our behalf. In addition, only Microsoft — and no third party — has the authority to clear Windows Defender detections.
With regards to AV companies, virtually all expressed concerns and frustrations with regards to AppEsteem’s “AV testing” initiative, which is seen by most in that sector as an attempt to force the consumption and automatic detection of items on the deceptor list. AppEsteem has stated that the AV testing is designed to “encourage AV efficiency by running comparative tests for detecting Deceptors and allowing Certified apps.” On its face, this statement clearly suggests that an AV would only be considered truly “efficient” if they detected so-called deceptors and allowed AppEtseem certified products as a matter of course (i.e., in an automated “consume and react” fashion). This largely contradicts AppEsteem’s other statements that they “recommend that security companies complete an independent review of any apps or services that we have designated as a Deceptor.,” and, only “[i]f they agree with our designation, we encourage them to detect, block, remove, or disable the app or service so it cannot harm consumers.” While these concepts are not wholly irreconcilable, the clear message as understood by AV companies is that the tests are being used as a forcing function, especially when it comes to detecting a “deceptor” whether an AV independently agrees or not.
- Double standards — inconsistent (and subjective) application of criteria –
- as applied to AppEsteem customers;
- as applied to security vendors;
Per AppEsteem, detection as a deceptor happens when a product engages in behaviors AppEsteem believes are “deceptive and risky behaviors that could harm consumers.” Further, “An app that violates one or more of our Deceptor Requirements will be designated a ‘Deceptor,’ published on our website, and reported to our security partners.” And finally, “we hold everybody accountable and build a level playing field…” This does not always seem to be the case, however.
With regards to certified versus “deceptor” products, the CSA has conducted a review of nearly all AppEsteem certified products and contrasted those products with products characterized as “deceptors” by AppEsteem. This exercise was done for the limited purpose of identifying whether standards seemed to differ when applied to products offered by AppEsteem customers versus products from companies who have not associated with AppEsteem. Whether any particular behavior was problematic or not was not a part of the assessment. In 20+ cases, AppEsteem customer products were left unscathed (and often enough, AppEsteem certified) while behaving identically or substantially similar to products deemed “deceptors” by AppEsteem. A consolidated version of the CSA’s comparison is included as Appendix 4.
Often, when confronted with this fact, AppEsteem notes that certified products (i.e., AppEsteem customer products) retain their certification for a full year and will not be held accountable during that year long period for violating deceptor criteria introduced since their certification. This, of course, raises the important question of how sound the deceptor program can be if some deceptor-level behaviors are hidden behind AppEsteem certifications. There is no more deceptive or risky behavior than that.
With regards to AV companies, much was brought to light during and then immediately following AppEsteem’s 2018 Affiliate Summit West meet-up in Las Vegas. Notably, AppEsteem’s presentation opened with the statement that, “[w]e hold everybody accountable and build a level playing field so clean monetizers can thrive,” however, when inquired about AppEsteem’s role in keeping AV companies accountable when they engage in “deceptor” behaviors, the response was that “holding AVs accountable is very tricky,” and “we rely on AVs to drive urgency, so [AppEsteem] won’t bash AVs.” It was further stated that the “cost of publishing AV deceptors” would be “blowing up” their program.
As noted above, in the week following the Las Vegas meet-up the CSA received a PDF purporting to show products from AV companies engaging in some of the very same behaviors as so-called “deceptors.” Upon receipt of the PDF, the CSA shared the materials with AppEsteem and its security partners. Some changes to AppEsteem ACRs ensued, but there has yet to be an AV product of consequence included in the deceptor list. This is notwithstanding the fact that specific examples had been shared with AppEsteem, and notwithstanding AppEsteem’s own repeated statements to the industry that it was running AV products against its criteria.
As part of its review, the CSA went through various AppEsteem materials and was reminded that in May 2017 AppEsteem communicated to its Security Partners about one particular behavior that AppEsteem calls out when engaged in by non-AVs. At that time, the message to the AV companies was that AppEsteem needed them to “clean up their own houses.” Now, approximately 11 months later, the CSA took screenshots of two of the three products depicted in AppEsteem’s materials. Nothing obvious has changed. Including the fact that neither product is listed as a deceptor by AppEsteem. A copy of the original May 2017 slide and April 2018 screenshots are included as Appendix 5.
- The program is lacking in adequate oversight and an appropriate dispute mechanism.
In the words of our respondents:
- “a lack of trust and a lot of concern around the AE program”
- “[deceptor program] is a divisive thing, really.”
- “Nobody likes deceptor program.”
- “Deceptor program just feels wrong on multiple levels.”
All indications through the CSA’s review process are that the deceptor program lacks oversight or an appropriate dispute mechanism. As most respondents were acutely aware, the deceptor criteria are ultimately determined and then interpreted by AppEsteem, and any appeals regarding a finding that a product is a “deceptor” must be lodged with AppEsteem and will be determined by them. Of additional concern is the fact that some respondents were under the mistaken impression that the CSA had endorsed AppEsteem and/or were providing oversight or dispute resolution for AppEsteem and the deceptor program. This is a notion that both the CSA and AppEsteem should dispel.
- The idea of a well-designed and run (and appropriately overseen) “deceptor program” which acts more as a source of “trusted leads” of problem software, is something AV companies reacted favorably to.
- AppEsteem’s certification program, and in particular its compliance-related services, are well regarded by numerous monetization companies.
Of the respondents currently working with AppEsteem, most, if not all, found the certification process, and especially the liaison services with regards to AV companies, quite valuable. While some were initially skeptical, most expressed having had a good experience overall, with a constructive back-and-forth in making their products compliant. In the case of companies converted into AppEsteem clients after initially being on the deceptor list, while all expressed distaste for the tactics used to get them to engage with AppEsteem, at least some are now willing to look past that and focus on the assistance they are now receiving from AppEsteem to engage in more compliant practices.
- Refocus efforts on certification
- Work with the CSA to devise consensus-built minbar criteria
- Balance violator identification and remediation
- Embrace oversight and dispute resolution
“Sell me on the merits of your certification program; don’t threaten me into being a customer.”
The above quote from one of our respondents is a perfect encapsulation of the sentiment shared by nearly all non-AV respondents. There is a clear, overwhelming desire on the part of the monetization companies that AppEsteem refocus on its original mission; namely, providing certification and outsourced compliance services. Our first proposal, therefore, is for AppEsteem to do just that – i.e., stop focusing on the divisive deceptor program, and focus instead on growing its certification and compliance business.
Given the AV respondents’ generally positive view on a more appropriately designed program, our second (not mutually exclusive) proposal is that AppEsteem give the CSA an opportunity to establish objective, “minbar” criteria through a consensus-based approach, complete with a feedback loop and fulsome industry participation. Requests for this sort of minimum objective criteria have already been received by the CSA from multiple sources, and this is an initiative the CSA is particularly well suited for. The CSA would welcome AppEsteem’s participation in identifying appropriate criteria, both initially and then on an ongoing basis.
Once a consensus-built set of criteria has been established, AppEsteem could adopt these criteria for its own purposes and/or potentially work hand-in-hand with the CSA in identifying potential violators of the minbar requirements. We would propose a more balanced approach to identifying violators, including a balanced review process and defined efforts to engage with software developers when deemed appropriate. Most importantly, being truly consensus-built would ultimately mean there is no need to threaten independent AV tests, as a true industry standard would be developed.
Finally, in any case, the CSA proposes that AppEsteem reconsider its posture regarding oversight of its programs and the provision of legitimate dispute resolution options by an independent non-profit association. We are aware of the non-profit being formed and incubated by AppEsteem itself. This organization was originally described as a non-profit being formed to provide “oversight” to AppEsteem. Any attempt by AppEsteem to create its own oversight body would only amplify the existing concerns regarding AppEsteem’s programs and behaviors, including the lack of oversight, lack of checks and balances, potential conflicts of interest, and more. In the week leading up to the publication of this Report, AppEsteem sent the CSA a communication confirming it is responsible for launching the non-profit (CleanApps.org), and that it continues to advise the organization and provide resources. Importantly, AppEsteem also clarified that it is only seeking “insight and input” from the organization, and does not intend for the organization to provide any oversight of AppEsteem or its programs. As such, vis a vis AppEsteem, the CleanApps.org organization appears to be performing the same role as AppEsteem’s existing “Steering Committee” and “Customer Committee,” namely, learning of certain initiatives and details in advance, and provided feedback. Indeed, the President of CleanApps.org, as well as all board members, are paying customers of AppEsteem. Consequently, the lack of oversight and legitimate dispute resolution mechanism persists.
To be clear, the CSA is not demanding or otherwise attempting to mandate any specific action(s) be taken by AppEsteem. That is entirely in its management’s discretion. The above are merely proposals shared by the CSA in a good faith effort to help guide AppEsteem back toward industry enhancing practices, and come with an open offer from the CSA to assist AppEsteem in any way it can with AppEsteem’s efforts to remediate and rehabilitate its programs.
- Appendix 1: com complaint regarding AppEsteem
- Appendix 2: Anonymous email regarding AppEsteem practices
- Appendix 3: AppEsteem prospecting email
- Appendix 4: Consolidated comparison summary, AppEsteem Certified vs. AppEsteem Deceptors
- Appendix 5: AV product behaviors, May 2017 vs. April 2018
About the Clean Software Alliance
The Clean Software Alliance (CSA) is a champion of sustainable, consumer-friendly practices within the software distribution ecosystem. This report is the work product of the CSA and the views expressed herein were prepared exclusively by the CSA and not any individual or officer, director or member of the CSA.
The CSA works to advance the interests of the software development community through the establishment and enforcement of guidelines, policies and technology tools that balance the software industry’s needs while preserving user choice and user control.
A 501(c)(6) nonprofit trade association, the CSA works inclusively across its constituents of online security vendors, software distribution & monetization firms, installer platform companies, browser providers, computer platform developers and others to find consumer-friendly solutions to the challenges of economically sustainable software distribution.