Anatomy of a Press Release
Today’s email inbox produced a wonderful little press release:
Positive Results for HomeoPet’s Storm Stress Presented at AVMA Conference in St. Louis
Westhampton Beach, NY – June 8th, 2012 – Dr. Kirsti Seskal, an internationally recognized triple boarded behaviorist, presented the results of her 2 year study on HomeoPet’s Storm Stress at the AVMA conference in St. Louis. Dr. Seskal’s study shows that Storm Stress, a 100% natural pet remedy alleviates fear and anxiety in pets during thunderstorms. Storm Stress achieved statistical significance over the placebo in two reporting periods, proving a very high safety margin with no adverse events reported during the two year study.
In other data collected over 5 years and reviewed by Dr. Seskal pet owners reported a 97% rate of improvement in pets treated for veterinary and grooming visits, being left home alone and general anxiety problems.
40% of dogs in the USA suffer from storm phobias, Storm Stress is a fast acting, non-sedating liquid remedy that promotes a sense of calm in pets with weather and noise related phobias.
Quote from Daniel Farrington:
“Established in 1994, HomeoPet has become the leading source for advanced natural stress products in the veterinary field, with products sold in thousands of veterinary clinics and stores in the US and Canada, and is now available in 8 countries around the world.”
Sounds impressive, doesn’t it? Lots of sciencey sounding language, and clearly positive results. But let’s take a closer look and break this down bit by bit:
Dr. Kirsti Seskal, an internationally recognized triple boarded behaviorist, presented the results of her 2 year study on HomeoPet’s Storm Stress at the AVMA conference in St. Louis.
Her results should speak for themselves, so it doesn’t really matter who Dr. Kirsti Seskal is. But I was curious nevertheless, since she’s “internationally recognized”. If so, that would be quite a trick; for Google’s record of the entire Internet does not contain even a single reference to that name. But let’s see what she presented at the AVMA conference in St. Louis. The 2011 conference of the American Veterinary Medical Association was indeed in St. Louis (the 2012 conference in San Diego has not yet taken place as of this writing), and we can check the online notes for this conference. Unfortunately, neither the Search, the Table of Contents, the Sponsors, nor the Author Index contain any mention of Kirsti Seskal or HomeoPet. This is not an encouraging start to our evaluation of the press release. Perhaps she was merely an attendee walking around and handing out flyers, which I suppose could be described as “presenting the results of a study”.
Dr. Seskal’s study shows that Storm Stress, a 100% natural pet remedy alleviates fear and anxiety in pets during thunderstorms. Storm Stress achieved statistical significance over the placebo in two reporting periods, proving a very high safety margin with no adverse events reported during the two year study.
I wonder what placebo is used for a homeopathic study. Homeopet publishes their ingredients online. The $29.99 bottle contains 5 ml of water, plus homeopathic quantities (meaning “spiritual essence”, but no actual molecules) of seventeen different compounds. Manufacturing must be nice when you need nothing more than a municipal water supply and only “spiritual essence” in your inventory.
Nevertheless, I don’t want to be argumentative. I would love to know more details about this alleged study was done. Mostly I’m curious to know what the placebo was.
In other data collected over 5 years and reviewed by Dr. Seskal pet owners reported a 97% rate of improvement in pets treated for veterinary and grooming visits, being left home alone and general anxiety problems.
And there we have it; the entire “study” consisted of nothing more than a review of customer testimonials. There is no mention of any blinding involved, which by itself invalidates the study. But just as bad is the reliance upon customer testimonials only. Entire books could be written on why this is invalid. What was the method of administration? Did it include any cuddling or calming language? How were the pets’ anxiety levels determined before and after the administration? What were the storm conditions before and after the administration? Did the pets have time to acclimate to the noise over the course of the administration? What other conditions may have affected the observers/owners, and what other conditions may have affected the pets? But this is pointless; we could go on and on.
If any of my analysis is in error, I invite correction from HomeoPet and/or the elusive Dr. Kirsti Seskal. Details of her presentation at AVMA would be especially appreciated, as well as its reception among the attendees. I’m most curious to know how, or if, she controlled for other factors in calming the pets, particularly the cuddling and calming language that are commonly used by pet owners and groomers attempting to calm nervous animals.
I do not expect such corrections to be forthcoming, but let’s knock on a homeopathic preparation of wood.
It sounds like the 5 years of data from customer reports is not the same as the 2 year study with a placebo. The paragraph is introduced with “In other data…”.
I’m more interested in the “statistical significance over the placebo in two reporting periods”, which sounds like classic cherry-picking of data. How many “reporting periods” were there that did not achieve significance? Presumably they’re using p=0.05 as the cutoff for statistical significance, and are not compensating for multiple comparisons, etc.
Here is information for you to look up.
Dr. Seksel presented at the IVBM International Veterinary Behaviourist Meeting. this was in front of some 200 specialists.
Research published this year in in the January edition of the Journal of Veterianry Behavior. This was work conducted in Lincoln University
Prof Daniel S. Mills BVSc PhD CBiol FIBiol FHEA CCAB Dip ECVBM-CA MRCVS
European & RCVS Recognised Specialist in Veterinary Behavioural Medicine
Dept of Biological Sciences
Dr. Seksel’s
Dr Kersti Seksel
BVSc (Hons) MRCVS MA (Hons) FACVSc DACVB CMAVA DECVBM-CA
Registered Veterinary Specialist, Behavioural Medicine
Sydney Animal Behaviour Service
55 Ethel Street
Seaforth
NSW 2092
Australia
Telehone: +612 9949 8511
Fax: + 612 9949 6364
http://www.sabs.com.au
4 years of studies were funded in the UK and the Australian study was 2 years long with multiple reporting periods and yes we did achieve stastical difference over placebo!
These are some of the most respected people in their field Maybe this will add some time for thought??
I think that you meant the research was published in January of 2011 not this year. Here is the abstract:
http://www.journalvetbehavior.com/article/PIIS1558787810001048/abstract
This study did not appear to have any bearing on whether or not the homeopathic preparations used had any efficacy.
Thank you daniela.
You say: “we did achieve stastical difference over placebo!”. By saying “we” are you indicating that you are part of the research team?
If so, can you give us more insight into what was presented at the IVB Meeting? Is there an abstract available on line? I was able to find a “call for papers” for the 8th International Veterinary Behaviourist Meeting in South Africa, which was advertized as being held in conjunction with the 30th World Veterinary Congress in October of 2011. Is this the meeting? Their program (pdf) does not list Dr. Seskal, nor any homeopathic presentations, but perhaps the IVBM has its own program?
tmac57 has already asked about the JVB publication. If that’s the correct abstract, then I agree that it does not lend any support to the effectiveness of homeopathy, nor does the study appear to be intended to do so. It looks to be a measure of how knowing that a subject is part of a placebo-controlled trial affects their subjective reporting of improvement, and how consistent their reporting is between trials and purported remedies.
And on purely a grammatical note, achieving an unspecified degree of statistical significance vs. placebo with an unstated, subjective effect size, an unstated population size, and an unstated total number of measured endpoints does not warrant the use of an exclamation point.
–James
So you spelled both her first name and her last name wrong in your press release? Really?
Perhaps Dr. Seskal is homeopathically famous.
There’s a Dr. Kersti Seksel who published a study “Kitten Socialization and Kitten Kindy” at the 2011 conference. Google Scholar can’t find anything by her related to homeopathic remedies.
Here’s the summary of their research
http://www.homeopet.com/research.html
They say they did “5 double blinded placebo controlled trials,” which would control for any cuddling or calming language, but as James said above they could’ve cherry-picked reporting periods, and their graph on this page shows 65% improvement with placebo vs. 70% improvement with their product. The axis is cut off at 60% to exaggerate the difference, and they don’t give the p-value.
“Manufacturing must be nice when you need nothing more than a municipal water supply and only ‘spiritual essence’ in your inventory.”
They need at least some quantity of the ingredients in order to dilute them.
By 70% improvement, I meant 70% of dogs showing improvement.
“They need at least some quantity of the ingredients in order to dilute them.”
No, not necessarily. They could just lend them and put them next to a quantity of water they would later use as “basic substance”. Nobody could ever find any proof that they actually owned the “diluted” ingredients.
Thank you Max! The reference accompanying the graph on the page you found is:
Cracknell NR, Mills DS. A double-blind placebo-controlled study into the efficacy of a homeopathic remedy for fear of firework noises in the dog (Canis familiaris). Vet J. In press
I found the abstract here.
It’s a paper published in: “The Veterinary Journal, Volume 177, Issue 1, July 2008, Pages 80-88″
Owners of 75 dogs were randomly and blindly assigned to placebo or homeopathic remedy groups, and reported baseline (at start of experiment) and final (at conclusion of experiment) assessments of “frequency and intensity” of “behavioral signs” of fear of fireworks. The placebo group reported improvement in 14/15 signs and the homeopathic group in 15/15.
While the HomeoPet.com page emphasizes that the improvement over baseline is significant, the abstract concludes that “there was no significant difference in the response seen between the two treatment groups.” (i.e. TFLN offered no benefit over placebo in this experiment.)
While the text of the HomeoPet.com page also incorrectly attributes the significant improvement over baseline to the effect of TFLN, the graph correctly attributes the effect to “TFLN + Behavior Training”
The press release misspelled Dr. Kersti Seksel’s name.
http://www.sabs.com.au/staff_profiles.htm
That would explain the lack of hits on Google.
Research like this on animals is tricky. So much is purely subjective–in the eye of the beholder (i.e., the owner). It will mimic similar “research” in humans.
That’s what the placebo and blinding are for. Also, it needs to be randomized to avoid giving all the kind and caring pet owners the treatment and giving the mean and abusive owners the placebo.
Also,it would help if their research were published in a peer reviewed journal with anything resembling quality. It looks like they are 15 years into some sort of research project…isn’t it time to publish?
The AVMA conference notes do mention Kersti Seksel, although her presentation there isn’t about homeopathy but about “Kitten Socialization and Kitten Kindy.” Maybe she mentioned her study then.
There were a couple of presentations on homeopathy by other presenters. Maybe she was a co-presenter.
The boarding place recommended some homepathic crap for calming the dog during a thunderstorm. She said it really worked, but you should ignore the instructions and give him loads and loads of it. I protested that there was no way that the stuff could work.
Then I looked more closely at it. At the suggested level? No, probably wouldn’t work. At her directions? Yeah, it probably would. It was 40 proof. And reading that page, this is probably the same ” Where animals are sensitive to the odor of the alcohol preservative”
lol…..I KNEW it’d be something like that….ie the dogs are calm because they are rotten drunk from the preservative NOT because of the homoeopathic ‘substances’ the the alcohol preservative is supposedly preserving.
Listerine mouthwash lists eucalyptol 0.092% and menthol 0.042% as active ingredients, and lists alcohol 21.6% under inactive ingredients.
http://www.drugs.com/drp/cool-mint-listerine-mouthrinse.html
I love the tone you achieved in this post! The sarcasm is gentle but absolutely scathing at the same time. What’s so really cruel about all this is that pets can’t speak for themselves and many genuine health problems may go untreated as a result of this kind of hocus-pocus.
Although it is obvious I should think, I would point out (to Max, especially) that no matter how the studies were done, it is all still Tooth Fairy Science.
http://www.skepdic.com/toothfairyscience.html
The tone is characteristically patronizing, yet one or both of the main criticisms – absence of the researcher and lack of blinding – were in error. The more you mock, the more it backfires when you’re wrong.
I did some volunteer hours at a vet clinic recently and one of the docs there was really into homeopathy. She mostly did acupuncture and I was interested in finding out if it actually worked, and if so, how it worked. I asked her (very nicely) what kind of results she had seen and where I could read about studies that were conducted and she got really defensive. She just kept saying the studies where online, but wouldn’t say where. I wasn’t trying to imply she was a fake or anything, she was a great vet, but she seemed a little too defensive to be totally confident in her work. She once put a needle on a dog’s nose that had a fever (I guess that’s supposed to relieve the fever) while I was giving him fluids and rubbing his belly and talking to him so he wouldn’t be so stressed.(Stress can increase body temp) When I checked him about 30 minutes later he still had a fever, but his temp had gone down about 1 degree F. The doctor of course said it was entirely due to the acupuncture…I however wasn’t convinced.
Last year during a controversy about establishing a goat farm in Arcata, this fake animal welfare organization “Animal Welfare Approved” weighed in, advocating homeopathic remedies for farm animals.
But not even the placebo effect would happen with goats, so how does one care about animal welfare by giving them water instead of antibiotics?
There’s a dialogue in the comments here:
http://www.arcataeye.com/2011/07/the-goat-chronicles-
Ahhh homeopathy… Did you know that drinking ocean water can prevent cancer, aids and most human affection?
There are indeed people with such affection taking bath in the sea…
I can tell you that, I did a study (5 long seconds) and I am a famous researcher ;-) !
The “Thundershirts” (as seen on TV and http://www.thundershirt.com) and similar brands cost between $25-$40, depending on the brand and store, and if it doesn’t work you can get a refund. A one-time purchase that doesn’t require repeated dosings and visits to the Vet. My sister and my cousin bought Thundershirts for their dogs, and they say they work. If my dogs’ Vet ever suggested homeopathic remedies I would find another Vet. In addition, many people don’t know that many prescriptions for animals can be purchased at any pharmacy for significantly less than the price charged by a Vet’s office.
I can see that some of the discussion regarding results are based on a basic “no-nowledge” to what statistical significant means. It takes at least a higher education in statistics, not to get fooled by people, who wants to fool you with numbers. Think about it – if homeopathy was a miracle, I am sure it would be used more in veterinary medicine, after all it is cheap and easy to use.
Let me explain it to you in a very simple way. If you have 50% improvement and it stands alone as a phrase, that in fact means that there is NO improvement at all!
Logic: in in veterinary medicine, the patient is not asked and the improvement is based on a second opinion, by the owners perception. If 50% of owners say there is no improvement and 50% say that there is an improvement, that means that there is a 50:50 chance to register an improvement by a second opinion holder. Think about it 50:50 chance is the same as a coin toss, when choosing who starts a football match. And as everybody knows, this is a matter of pure coincidence.
No – numbers need to be calculated in special ways to be referred to as statistical significant. Don´t be fooled by people who are religious in special ways of veterinary treatment or try to sell you a concept. Ask an expert, who knows if the numbers are calculated in the right way.
Another example: My dog likes beans. A survey with one dog would show that 100% of all dogs in my survey like beans. Smart ;) LOL
PS: All the significant references/links in this discussion are not existing anymore – I wonder why? ;)