Trust Must Be Measured: A Conversation with Andreas Clementi, Founder and CEO of AV-Comparatives.

In this new English episode of My Precious Data, cybersecurity expert Eddy Willems sits down with longtime colleague and friend Andreas Clementi, founder of AV-Comparatives, the independent organisation behind some of the most respected security product tests worldwide.

For decades, AV-Comparatives has been the silent benchmark behind the cybersecurity industry. Vendors build products. Marketing teams make claims. But independent testing determines what truly works.

Eddy and Andreas go back many years, to a time when antivirus testing was less structured and transparency in performance evaluation was still evolving. Together, they reflect on how independent testing became a cornerstone of trust in the security ecosystem.

“Independent testing is not about attacking vendors, it’s about improving protection for users,” says Andreas Clementi.

“Without transparency and objective testing, trust in cybersecurity would simply collapse,” adds Eddy Willems.

The conversation dives into:

• The origins of AV-Comparatives
• The evolution of malware testing methodologies
• False positives, performance testing and real-world scenarios
• The pressure and responsibility of independent testing
• The future of security validation in an AI-driven world

“Testing must evolve as threats evolve. Otherwise we measure yesterday’s problems, not tomorrow’s risks.” – Andreas Clementi

An insightful and candid discussion about trust, integrity and the often unseen backbone of the cybersecurity industry.

Transcript

SPEAKER_00 0:04

Welcome to My Precious Data, the podcast by Eddie Willems, where we explore the world of cybercrime and incidents affecting our society. In this series, we engage in conversations with Eddie Willems, security evangelist, globally recognized cybersecurity expert, and international keynote speaker. Eddie also dives into conversations with fellow experts to share insights and explore new perspectives. Together, we highlight the latest security trends and discuss preventive measures to help avoid these kinds of problems.

SPEAKER_01 0:42

Good morning, good evening, good night. Whenever you're listening, welcome back to my precious data. Today's guest is someone I've known for many years, and someone who plays a very unique and important role in the cybersecurity ecosystem. He's the founder of AVE Comparatives, one of the world's most respected independent security testing organizations. Andreas Clementi. Andreas, it's great to have you on the podcast. Hello.

SPEAKER_02 1:14

Hi.

SPEAKER_01 1:15

Hey. Andreas, first of all, uh, where are we over here? You know?

SPEAKER_02 1:22

Um, we are in Innsbruck, where AV Comparatives is uh based. Um, we are recording this while we are having our annual AV Comparatives Summit. And this year we are also hosting the Caro workshop.

SPEAKER_01 1:36

Yeah. Yes, and I'm also over here actually for that uh uh summit and the uh caro workshop. So you and I have known each other for quite some time now. Do you remember when and where we first met? Do you remember something? Because it's for me a little bit blurry.

SPEAKER_02 1:55

Yes, I think we first met uh 2004 in Luxembourg at the ICA conference. Oh yeah, yeah, yeah. Yeah, yeah. This was my very first uh conference because usually I do not attend conferences, and this was my very first one because I talked ICA is uh very academic and uh scientific and not so marketing driven.

SPEAKER_01 2:21

So true, yeah, yeah, yeah. No, I remember it as well. Yeah, okay. Um yeah, and actually for the for the people who are listening, uh, you know, this is really rare that you can hear Andreas giving an interview. So I think this is remarkable that he uh said yes to be on my podcast.

SPEAKER_02 2:44

You know, I'm I'm very shy and I don't really like to be in the spot on this in the spotlight. Uh-huh. So this feels a little bit unusual for me.

SPEAKER_01 2:52

Yeah, but that makes sense, no problem at all. So, for the listeners who may not know your background, how did you personally get into cybersecurity and what led you to focus on independent testing?

SPEAKER_02 3:06

So it started very early for me. Uh, so when I first got my uh first personal computer in 1993, I got infected with a virus November 17. That made me curious about the topic, and of course, I wanted to protect myself, but I was keep getting infected. So I was reading uh magazines to find out what is the best uh antivirus, but um every magazine said something different. Typical. I did not really believe what they write, so I talked, uh I have to find out myself. So I started uh researching the topic, let's say, and testing myself to find out what is the right product to use. Exactly.

SPEAKER_01 3:58

Yeah, yeah. Well, yeah, I I totally uh it it's it's it's it's typical. I think it's uh also my journey started somewhere with something like that. So yeah. So let's go to the birth of A V comparatives. So, well, you you said it already. Uh so what motivated you to start with AV comparatives? Because, of course, you told already that yeah, why you started with it more or less, but uh what is something else, maybe?

SPEAKER_02 4:30

So of course it was not a business idea. I never wanted to to do a business out of it, it was really just a hobby for myself. Yeah. Um, I tested the products uh well for myself, actually, actually, and also for my friends uh at school. So I wanted to understand uh what is true and what not. So I tested myself and um I tried to figure out what is the best way to test antivirus products, and because uh people um were interested in the results because I posted them online, it uh started to get my focus hobby. Yeah, and yeah, at some point it uh went very big, and I had to create a V comparative S.

SPEAKER_01 5:23

Exactly, yeah, okay, makes sense. So at the time the antivirus industry was already growing fast. Why did you feel there was a need for independent testing?

SPEAKER_02 5:34

Yeah, so as the industry got uh bigger and bigger, also the marketing got bigger. Every vendor claimed to be the best. And for for me and also for users in general, it became very difficult to understand uh what is true and what not. So we're saying uh who is really the best product to use. So that's why I think that uh testing in general is very important. Of course, independent testing is very important, of course, and if done right, it gives uh a data point to the users so that uh they can understand better what is uh good and what not, where are the weaknesses of the products, where are the strengths. Yeah, so yeah, it's information.

SPEAKER_01 6:18

Yeah, true, yeah. So, was it difficult to convince vendors to participate in transparent and comparative testing in the beginning?

SPEAKER_02 6:28

So, in the beginning, I just spoke all the products myself, and uh I did not work with the vendors directly in that sense. Uh I just tested them. Yeah, um, the environment was also very different. Um, you could find uh malware easily online, I mean for research purposes. Um there were also much uh fewer samples out there. So the goal was to have uh the big the biggest collection, yeah. Uh so to have a very representative uh test set to use. Uh there were also other uh good testing labs, like I like the virus test center in Hamburg very much. Yeah, yeah, yeah, yeah. Um, and this was very academic and uh research driven. Yeah. So in general, yeah, the atmosphere was very research-driven back then.

SPEAKER_01 7:26

Exactly. Okay, so independence is at the core of AV comparatives, I think. So, how hard is it to remain truly independent in an industry driven by marketing and competition?

SPEAKER_02 7:41

Yeah, so independence is not something that you reach one time and that's it, you have to work on it uh all the time. And the industry is, of course, very much driven by marketing and competition, which creates a lot of pressure. Uh yeah. So vendors want good results, yeah, and nobody wants to look bad in the public. So the challenge is to stay focused uh on facts and not on expectations. Exactly.

SPEAKER_01 8:11

Yeah. So what does fair and scientific testing really mean in practice?

SPEAKER_02 8:19

Um, it means to be consistent, honest, up to date. Uh scientific means, of course, that the results are based on observation and uh data, so not opinions or subjective uh factors. So we document everything, uh how it happens and when it happens. Uh, and if you read our test reports, you will also find the limitations uh of the test. So we are open about uh how we test and what are the limitations so that the readers can understand uh what we did and why we did.

SPEAKER_01 9:00

Indeed. Yeah, yeah. So have you ever faced pressure, direct or indirect, to influence test results? It's a tricky question, I know.

SPEAKER_02 9:12

Yes, this happened over the years. So there were uh legal threats, uh, but none of them succeed. Um, there were also attempts to discredit uh our work, or also on a personal level, or to make the research more difficult or almost impossible, but uh yeah, they did not succeed. Good, good to hear. And it's also important to say it's of course not easy to stand against companies with large communication budgets. Yeah, so we are just a small company, but we do the right things, so they cannot succeed. So our response was always uh to be stricter with the methodology and uh to document even more than before.

SPEAKER_01 10:01

Okay, mm-hmm. Interesting. So let's go to the evolution of malware and protection. So uh you've had the front row seed to the evolution of threads. How has malware changed from the early virus days to today's ransomware and advanced attacks?

SPEAKER_02 10:21

So in the early days, there were two worlds, I would say. So there were the researchers on one side which tried to protect the users, and there were underground communities which were experimenting with new techniques. So in the past, there was this was driven by curiosity and creativity, I would say. People brought Malver to see what is possible or to challenge each other on a technical level, or sometimes just for fun. Yeah. So I don't know if you remember, there were even underground magazines. Yes, I remember them. And yeah, it was an interesting time. So in the past, um young people just created malware to show off or to show what is possible, and today it's mainly about crime and profit.

SPEAKER_01 11:14

Yeah.

SPEAKER_02 11:15

Yeah, yeah, true. So um early malware was uh about technical challenge, and today it's more about ransomware, data theft, espionage. So it's to make a profit nowadays. Yeah, exactly.

SPEAKER_01 11:32

Are these type of threats today that you feel are still underestimated by users or organizations? Are there any threats uh underestimated, you think?

SPEAKER_02 11:43

I think the most um underestimated threat is social engineering. Exactly. Because it lowers the uh defenses psychologically, and attackers use fear, urgency, and trust to bypass rational thinking.

SPEAKER_01 11:59

Yeah, I'll I'll love to hear this.

SPEAKER_02 12:02

Another area is also the leaving of the land uh attacks, which use normal system tools. Uh supply chain attacks are also underestimated. So when trusted software is compromised, many victims are affected. And I think that uh this is why our advanced threat protection test is so useful, because it simulates uh complex scenarios. We even offer it as a free add-on for uh the consumer tests, main test series, for example, but not every vendor participates in it uh at the moment. And regarding the organizations, um, I think that many organizations overestimate what products alone can do because technology alone cannot replace awareness updates and backups, of course.

SPEAKER_01 12:53

Completely agree. So let's go to the measuring of real protection. I I think one of the biggest challenges in security is measuring real-world protection. What do users often misunderstand when they look at test results?

SPEAKER_02 13:13

Um, the the biggest misunderstanding is that they think that one number uh tells the world story. Um we have defined very easy to understand uh scoring models, at least, in my opinion, they are very easy to understand. For example, we have uh in the consumer main test uh series um the standard rating, so standard, advanced, advanced plus and tested, and uh reaching the standard rating is already a very strong result, but this is not very clear to the readers. So advanced and advanced plus were introduced only as incentive uh for improvement, but marketing often promotes only the highest awards and ignores standard completely. But like I said, standard is already a very good uh result. What is also to be said, a test is of course not the universal true, it's just uh a snapshot of a specific scenario at a given time. Um people also forget uh side effects like false positives or impact on system performance. So real protection is about uh balance. And of course, uh, because I said it's uh a snapshot at a given time, uh the test should be evaluated over time. Yeah, so looking at uh trends, not at rankings, it's not about reaching 100% in a test, it's more reducing the risk. It's not about you have to reach 100%. And this is why I recommend everyone to read the methodology in detail. Do not look only at one testing lap. Of course, they can look at our testing lab, but they also should look at other testing laps and different tests because the more data and information you have, the better picture you get.

SPEAKER_01 15:19

I totally agree.

SPEAKER_02 15:21

And maybe just just to make it more even more clear, many users treat test results like sport rankings.

SPEAKER_01 15:29

Oh, yeah.

SPEAKER_02 15:30

So they look who is number one and who is number two, but this is not uh what it is about. The real goal is not to declare a winner, it's more to show the positive and negative aspects of the product and to help them to make an informed decision.

SPEAKER_01 15:47

Yeah, yeah. Uh that's unbelievable how much is to say about detection and tests and so is detection rate still the most important, Patrick, or has that changed?

SPEAKER_02 16:02

Um, detection rate is still important, but not enough, of course. So attacks today are more um a chain of action. So what matters is how the products behave during an attack. So, like I said before, fights positives and performance are also very important aspects, as well as usability and response speed, for example. So we need a holistic view.

SPEAKER_01 16:29

Exactly. Okay, uh, do you think users and enterprises know how to properly interpret security test reports?

SPEAKER_02 16:39

Some do. Many may not, but this is understandable.

SPEAKER_01 16:44

Yeah.

SPEAKER_02 16:45

So I think uh a common mistake is uh stopping at the final score without reading conditions and limitations of tests. The results are often just used uh as marketing instead of technical guidance.

SPEAKER_01 17:02

Yeah.

SPEAKER_02 17:03

So reports, test reports should give uh a guide, maybe even narrow down decisions, but not replace thinking and own evaluations. And enterprises should look at vendor scores over time and across different test types, because like I say, you cannot look at just one number. And I repeat, I really recommend looking at more than one testing lab. And yeah, interpretation is improving, but education is still needed.

SPEAKER_01 17:39

Yeah, I also completely agree. Um AI, automation and the modern security, that's another thing, of course. So AI is now used by both attackers and defenders. How has this impacted the way you test security products?

SPEAKER_02 17:58

So AI is a new term which marketing really loves. Um but it's not really some it's not something really completely new. So machine learning, which is a super part of uh AI, is being used in security products in many many years, even decades. So but um what changed is that the scale and the dynamics are also thanks to improved computing speed, but it's not something completely new. So there's also deep learning which uh some products have. And of course, AI is being used in certain parts of uh the products. But yeah.

SPEAKER_01 18:50

Yeah, no, that's right. I think a lot of users don't know that AI is being used inside the products already.

SPEAKER_02 18:58

But for us, uh the principle remains the same. We just uh expose the products to the real attacks and observe the results. Uh as you know, uh in the past there was also the cloud, which was something new, which uh changed a little bit uh how testing has to be done. Um so we have of course to adapt constantly, and but we also need to protect parts of the methodology so that products do not uh optimize for the test. But we try to be as transparent as possible to still deliver fair results. And in the end, like I said, we measure just uh the outcomes we observe, we describe, and we look whether a product prevents damage, no matter how.

SPEAKER_01 19:53

Okay. Do you believe AI will make independent testing more difficult or more important than ever?

SPEAKER_02 20:02

Um both, but not in a new way. So, like I say, cloud protection already made systems very dynamic. AI just increases even more the marketing claims and complexity, which users cannot verify themselves, and this is where independent testing becomes more important.

SPEAKER_01 20:24

Um yeah, let's have a look to the security industry itself. So you work with almost all major security vendors. How has the cybersecurity industry itself changed over the years?

SPEAKER_02 20:39

The industry changed a lot. So in the early days, many companies were founded by small groups of engineers and researchers. They did it mainly driven by curiosity and the idea of protecting users. So it felt uh more technical and academic than commercial. Um, back then it was for me also common to communicate directly with the founders and developers. So the discussions were very uh technical and open. And today the industry is much more business-driven. Okay, so yeah, companies are now large organizations with marketing, sales team, strong competition, and marketing and branding plays a much bigger role today than in the past. And in my opinion, this also increases the need for independent testing to compare the real behavior and maybe even help to reduce the complexity for decision makers. But to be honest, I also feel a little bit nostalgic because many early pioneers, the founders, they are now starting to leave or to. retire and this leaves a gap. So in in the early days, all those founders and uh developers, those were really people which were extremely focused. They were oriented toward details and were deeply passionate. So like I'm passionate for testing, they were passionate about their products and protecting the users. So they could spend endless hours improving things because they cared about the topic on a on a different level than today. And I sometimes really miss that atmosphere.

SPEAKER_01 22:42

I believe you like that uh also actually yeah so actually it answered more or less my next question do you think vendors today are still driven by protection or more by marketing claims?

SPEAKER_02 22:56

Vendors in general I would say it's uh it's a mix.

SPEAKER_01 22:59

Yeah.

SPEAKER_02 23:00

So the technical people care of course about protection but competition and positioning out there is much louder than before.

SPEAKER_01 23:10

Yeah yeah other are much more companies as well of course so is there something the industry collectively does wrong in your opinion yes oversimplification okay so security is very complex but it's presented as if one product can solve everything another issue is um that there is too much focus placed on being the number one or having the best marketing claim this turns security into a competition of slogans and not so much on discussing about the real risk and real protection.

SPEAKER_02 23:56

And yeah when they talk about technology they do not focus enough on people and processes so many incidents do not happen because a product failed I would say but because of human behavior missing updates or the procedures are not the best.

SPEAKER_01 24:21

So security is about reducing risks but it's not about perfection that you cannot reach perfection security you you cannot buy security it needs to be lived and integrated into the processes exactly wow these answers are so good I hope everybody is listening to this podcast because this is for me fantastic um very interesting answers.

SPEAKER_02 24:53

Yeah some memorable moments and stories without naming names um what's one of the most surprising or unusual findings you've encountered during testing um there were of course many different things one example which comes to mind is that during Android testing there was one product uh which had uh a password protection uh implemented but you could enter any PIN code and it got unlocked so there was no um security feature implemented correctly so this shows that a small mistake because it's just one line of code which was uh wrong of course can undermine the whole security feature then another surprise is which happens several times over the years where uh there are companies which uh want to get tested and they think that they have a very revolutionary or the perfect product which now solves all the problems and then we test it and we show them that there are major flaws and uh errors and it does not work in reality and this then opens their eyes or maybe the eyes of the investors that they have uh behind it. So yes we had quite a few uh surprises looking under the hood but uh yeah bad ones but also a few good ones of course let's go further with personal reflection um after all these years what still surprises you about cybersecurity um sometimes it can be surprising where you need cybersecurity these days um so 20 years ago no one would have talked that they need uh an antivirus on their mobile phone for example so yeah it uh it really changes where cybersecurity needs to be implemented uh these days because everything is interconnected but if we look uh very closely uh the same patterns repeat uh every time both in extortion attempts but also in how you try to defend against them but the marketing language of course also changes but the logic stays the same so like I said all promises must always be checked against reality exactly what do you believe now about cybersecurity products that you didn't believe when you started A V comparatives yeah so also a difficult one I think not so difficult but yeah okay um so when I started I was uh very skeptical about the products and their claims uh in my opinion much of what was written in magazines and marketing did not and does not reflect reality um but over the years I I learned that some of the skepticism was uh justified more than you would expect but I but I also learned that security uh is very difficult yeah so like I said products cannot be perfect they are about uh reducing risk in complex and uh different environments so even good products will sometimes fail and sometimes then excel other products but anyway um I trust uh data more than promises of course I trust my data more than anything else and the long term results are what uh you have to look at and not uh at single claims okay if you were starting your career today would you choose the same path again um that's this is now a difficult question okay um because this was never really planned like I said um it was a hobby um only when it became bigger uh for logistical reasons and also for legal reasons because you are handling here multi-billion dollar companies uh I had to create uh a company out of it so I never wanted uh to work in the cybersecurity space in the classic sense but as you know I have many interests in many different topics and I always wanted to understand how things work and uh verify whether claims are true and this is how I think so yes likely I will choose the same path because it's my hobby it's what I like to do it's my passion yeah so yeah so I tested for myself first because I wanted to know what was true I shared the results because I believe the data was useful and what matters is that results are honest and based on observation. What today I will do different is to improve how it's being communicated outside.

SPEAKER_01 30:52

Oh yeah because um in the past we have done almost no marketing ourselves it was just uh other people sharing our work and uh distributing it but now we are trying to improve uh because reaching more people is very important because then they are also informed and can take decisions so my mindset would be will remain the same probably so being curious very skeptic and uh independent that will not change exactly then doing your hobby it's it's it's like me I I actually I always did my yeah hobby actually I never worked you know I always did my hobby still now I'm doing yeah my hobby this podcast is a hobby but it's my work more or less as well and anyway um now we coming uh always to something like a tricky thing as well for me because you know I've most of the time I send a couple of questions out to you know my participants in this podcast but then I always ask you there is always a one million dollar question as we call it and that's a a question I don't know from before and I always ask then my participant by Andreas if you could ask me one question what would it be so I have been talked about it so you have been in the industry for many many years and after all your years in this industry how do you personally decide what to trust in cybersecurity when there are so many strong claims and so much noise well I I I think you explained also a little bit about it because I always also looked at tests from before so if I was beginning in the industry I was trying to find out if something is is something already tested they everybody's claiming something so I tried to find out at that moment and that wasn't the case because in the beginning there was no testing at all and there were some tests from magazines which are in my opinion completely rubbish because they didn't do it well they only tried like with 10 samples and which are you know even not being well you know sometimes there were even not malware being tested as being a malware um so that's part of the thing I always wanted to find out why are they doing that so I always was looking forward to when the real test comp organizations or test companies are start are are starting and and and you were one of the first ones um and I was looking very well to that one so for me it was also something very important. On the second hand of course it's very very very difficult to test products it's unbelievable what you are doing because everybody has his own habits and own things he wants to do with the product itself. Of course they need to protect you and they need to protect your business but there are so many businesses one yeah you can try to test it but it's sometimes still difficult I think uh of course it's very difficult to say that everything is 100% you know that everything is can protect you 100% so in that way it's very difficult. Um yeah for me it's always been a tricky one and I tried also to look around me and there are so many companies these days much more than before and it's now even much more problematic I think because in the beginning yeah there were only like 10 to 20 products and now you have well much more well if you look to the whole security uh ecosystem and they all claim the same thing um oh my god no and that shouldn't be the case because you have products for this you have products for that and you need to make your right choice and it's something you need to be well and therefore you need to have antivirus testing labs like yours and you should do in my opinion also your own tests a little bit together with the other test results. So that's how I look to it so it's like a mixture of everything but you should do your own test more or less as well uh and and and and that's it. Yeah but it's a nice question because it's uh it's also not easy to to answer that actually but uh I like it okay uh anyway I think it also was evolving of course because in the beginning we were only viruses and and then uh and a couple of other welles and now you have so many other threats as well so it makes it difficult for everybody. Yeah okay unfortunately we are coming more or less to the end of this podcast um I think this was a wonderful podcast because we've talked about a lot of interesting things a lot of uh people which are listening to this podcast will learn a lot about it I think uh because even well yeah you know testing is not so known to the to to to well to every public it depends from public to public you know um there are a lot of users looking at it like you said like yeah who's winning the battle a little bit but it's still tricky um do you think we'll see each other again soon I don't know during a conference over there um I think not very soon because I don't attend that many conferences uh but very likely I think we will meet each other at the Marospoletin in October and of course you're always welcome also next year to come to our summit here in Innsbruck aha oh yeah this is noted by this and all the listeners are uh have heard this you know okay thank you for this I have one one one final question maybe is there one message you'd like to leave to our question uh our listeners uh about cybersecurity trust or independent evaluation some last thing you want to say my message would be stay curious and think critically do not accept claims just because they sound convincing look for evidence um trust should be based on long-term behavior and not on slogans and independent evaluations can help to bring some clarity but everyone like I said should learn to question and understand what they use exactly I personally you were taking this out of my mouth as well it's more or less my answer I wanted to say during my question actually okay Andreas thank you really for this conversation um and and for the work you've been doing all these years to bring transparency and trust into cybersecurity I think this is very very valuable I wish you a cyber safe continuation of your day and I really look forward to crossing paths again soon. To all our listeners thank you for tuning in to my precious data.

SPEAKER_00 38:59

Until next time stay safe stay critical and keep your data precious if you have any comments questions or suggestions feel free to email us at podcast at wavci.com for more information about lectures or keynotes please visit wavci.com see you next time