Automated Valuation Metrics
Earlier this year, Lee Kennedy appeared with Matthew Blake on the HousingWire Daily podcast, Houses in Motion:
They covered a number of topics in valuations, from iBuying to AVMs, including:
FHFA, the oversight agency for Fannie Mae and Freddie Mac, published a Request for Input on December 28, 2020. The RFI covered Appraisal-Related Policies, Practices and Processes. AVMetrics put forth a response including several pages and several exhibits making the case for using AVMs responsibly and effectively in a Model Preference Table®. Here is the Executive Summary:
The lynchpin to many of the appraisal alternatives is an Automated Valuation Model, a subject which AVMetrics has studied assiduously and relentlessly for more than 15 years. We point out that even an excellent AVM can be improved by the use of a Model Preference Table. MPTs enable better accuracy, fewer “no hits” and fewer overvaluations.
We also suggest an escalated focus on AVM testing, and we use our own research and citations of OCC Interagency Guidelines to emphasize the importance of testing to effectively use AVMs. We suggest that an “FSD Analysis” like the one we describe reduces risk by avoiding higher risk circumstances for using an AVM.
We suggest that the implementation of a universal MPT by the Enterprises will improve the collateral tools available and reduce the risk of manipulation by lenders. We also believe that a universal MPT can help redeploy appraisers to their highest and best use: the qualitative aspects of appraisal work. Our suggestion is that the GSEs endeavor to make the increased use of AVMs a benefit to appraisers, increasing their value-added and bringing them along in the transition.
AVMetrics’ full response is available here:
The AVMNews sat down with our publisher Lee Kennedy to discuss trends in the industry.
AVMNews: Lee, as the Managing Director at AVMetrics, you’re sitting at the center of the Automated Valuation Model (AVM) industry. What changes have you seen recently?
Lee: There’s a lot going on. We see firsthand how the evolution of the technology has affected the sector dramatically. The availability of data and the decline in costs of storage and computing power have opened the doors to new competition. We see new entrants using new techniques and built by fresh faces. We still have a number of large players offering well-established AVMs. But, we also see the larger players retiring some of their older models. The established AVM players have responded in some cases by raising their game, and in other cases, by buying their upstart rivals. So, we’ve seen increased competition and increased consolidation at the same time.
And, it’s true that the tools keep getting better. It’s not evenly distributed, but on average they continue to do a better and better job.
AVMNews: In what ways do AVMs continue to get better?
Lee: AVMetrics has been conducting contemporaneous AVM testing for over a decade now, and we have many quantitative metrics showing how much better AVMs are getting. Specifically, we run statistical analysis around the comparison of AVM estimates to sales prices that are unknown to the models. We have seen increases in model accuracy rates measured by percentage of predicted error (PPE), mean absolute error (MAE) and a host of other metrics. Models are getting better at predicting sale prices and when they miss, they don’t miss by as much as they used to.
AVMNews: What about on the regulatory side?
Lee: There is always a lot going on. The regulatory environment has eased in the last two years reflecting a whole new attitude in Washington, D.C. – one that is more open to input and more interested in streamlining. Take, for instance, the 2018 Treasury report that focuses on advancing technologies (See “A Financial System That Creates Economic Opportunities”).
Last November, I was at a key stakeholder forum for the Appraisal Subcommittee (ASC). One area of focus was harmonizing appraisal requirements across agencies. Another major focus was how to effectively employ new tools in support of the appraisal industry, including the growth of Alternative Valuation Products that utilize AVMs.
AVMNews: I know that you also wrote a letter to the Federal Finance Institutions Examination Council (FFIEC) about raising the de minimis threshold, below which some lending guidelines would NOT require an appraisal. This year in July they elected to change the de minimus threshold from $250,000 to $400,000 for residential housing. What are your thoughts?
Lee: Well, I think that the question everyone is struggling with is “What does the future hold for appraisers and AVMs?” Obviously, the field of appraisers is shrinking, and AVMs are economical, faster and improving. How is this going to play out?
First, my strong feeling is that appraisers are a valuable and limited resource, and we need to employ them at their highest and best use. Trying to be a “manual AVM” is not their highest and best use. Their expertise should be focused on the qualitative aspects of the valuation process such as condition, market and locational influences, not the quantitative (facts) such as bed and bath counts. Models do not capture and analyze the qualitative aspects of a property very well.
Several companies are developing ways of merging the robust data processing capabilities of an AVM with the qualitative assessment skills of appraisers. Today, these products typically use an AVM at their core and then satisfy additional FFIEC evaluation criteria (physical property condition, market and location influences) with an additional service. For example, the lender can wrap a Property Condition Report (PCR) around the AVM and reconcile that data in support of a Home Equity Line of Credit (HELOC) lending decision. This type of hybrid product offering is on the track that we’re headed down. Many AMCs and software developers have already created these types of products for proprietary use or for use on multiple platforms.
AVMNews: AVMs were supposed to take over the world. Can you tell us what happened?
Lee: Well, the Financial Crisis is one thing that happened. Lawsuits ensued, and everyone got a lot more conservative. And, the success of AVMs developed into hype that was obviously unrealistic. But, AVMs are starting to gain traction again. We are answering a lot more calls from lenders who want help implementing AVMs in their origination processes. They typically need our help with policies and procedures to stay on the right side of the Office of the Comptroller of the Currency (OCC) regulations, and so in the last year, we’ve done training at several banks.
Everyone is quick to point out that AVMs are not infallible, but AVMs are pretty incredible tools when you consider their speed, accuracy, cost and scalability. And, they are getting more impressive. Behind the curtain the models are using neural networks and machine learning algorithms. Some use creative techniques to adjust prices conditionally in response to situational or temporary conditions. We test them and talk to their developers, and we can see how that creativity translates into improved performance.
AVMNews: You consult to litigants about the use of AVMs in lawsuits. How do you think legal decisions and risk will affect the use of AVMs?
Lee: This is an area of our business, litigation support, where I am restricted from saying very much. It has been and continues to be an enlightening experience as some of the best minds are involved in all aspects of collateral valuation and the “Experts” are truly that… experts in their fields as econometricians, statisticians, appraisers, modelers, etc.… It is also very interesting with over 50 cases behind us now, to get a look behind the legal system curtain and how all of that works. Therefore, I want to emphasize that my comments for our interview are in the context of contemporaneous AVMs that were tested during the time period shown here and not a retrospective AVM that was looking back to these time periods.
AVMNews: AVMetrics now publishes the AVM News – how did that come about?
Lee: As you and the many subscribers know, Perry Minus of Wells Fargo started that publication as a labor of love over a decade ago. When he retired recently, he asked if I would take over as the publisher. We were honored to be trusted with his creation, and we see it as a way to be good citizens and contribute to the industry as a whole.
AVMNews: I encourage anyone interested in receiving the quarterly newsletter for free to go to http://eepurl.com/cni8Db
The AVMNews is a quarterly newsletter that is a compilation of interesting and noteworthy articles, news items and press releases that are relevant to the AVM industry. Published by AVMetrics, the AVMNews endeavors to educate the industry and share knowledge about Automated Valuation Models for the betterment of everyone involved.
As always, changes are coming to the valuation industry. These changes have been germinating in government and industry for a long time, but they’ve made progress in the last year, and I believe that they’re likely to emerge sometime this year. I expect that we may see more regulatory changes liberalizing the use of AVMs soon.
I think that you’ll come to that same conclusion, too, if I share a couple milestones that I’ve observed and put them together with some insights I’ve gathered from talking to industry leaders.
The first milestone I will highlight was the July 2018 Financial System report by Secretary Mnuchin, which is consistent with the administration’s new attitude towards regulation. The report is far-reaching, and it includes thoughtful commentary about the uses of AVMs (see, for example, page 103-106). It recommends updating FIRREA appraisal requirements to accommodate increased usage of AVMs and hybrids. It also advocates for increased monitoring of AVMs and the application of rigorous market standards. And, it recommends focusing the use of AVMs and hybrids on loan programs with other mitigating risk factors.
The next milestone I will highlight was the proposed change in the de minimis threshold that was put out for comment in November of last year. The change would raise the threshold below which a residential mortgage could be originated with an evaluation, utilizing an AVM in lieu of a traditional appraisal. It would be raised from $250,000 to $400,000.
To those milestones I would add a third data point. Last November I attended the Appraisal Subcommittee roundtable entitled: “The Evolving Real Estate Valuation Landscape.” As part of the of the Federal Financial Institutions Examination Council, the roundtable brought together industry representatives and government officials (see the table below) to discuss real estate valuation.
The day was split into two sessions; the morning and afternoon sessions each began with a panel of industry experts who addressed a series of prepared questions. In addition, there was a roundtable discussion focused on quotes from the July 2018 Financial System report referenced above.
The topic for the morning discussion was “Harmonizing Real Estate Valuation Requirements Across the Federal Government.” This session focused on identifying various federal appraisal statutory and regulatory requirements and exploring opportunities to harmonize those requirements, e.g., VA, FHA, and FHFA all having differing valuation requirements and standards.
The afternoon panel discussion topic was; “The Evolution of Real Estate Valuation” which focused on evolving valuation needs in commercial and mortgage lending. A key area of this session was focused on Alternative Valuation Products inclusive of AVM’s and their increasing used by lenders and the secondary market.
The roundtable discussion started with quotes about AVMs and hybrid valuation products and focused on standards. The group also contemplated how alternative valuation techniques can impact quality and mitigate risk. Finally, one quote that focused on speeding the adoption of technology was discussed.
As I write this six months later, I see the pieces of the puzzle coming together. Obviously, there is momentum behind the increased usage of AVMs, for their independence, increasing accuracy, speed and efficiency. But there is also an implicit concern to avoid opening the door to more risk. I see this being expressed by talk about “standards,” alternative products, such as “hybrids” and increased monitoring.
As I have written elsewhere, I welcome changes that make better use of our valuable and limited resources, namely the appraisers themselves. As AVM quality improves and the number of appraisers shrinks, we should encourage appraisers to be focused on their highest and best use. Their expertise should be focused on the complex, qualitative aspects of property valuation such as the property condition and market and locational influences. They should also be focused on performing complex valuation assignments in non-homogeneous markets. Trying to be a “manual AVM” is not the highest and best use of a highly qualified appraiser, and I expect that Treasury, the FDIC and legislators are moving in this same direction.
Lee Kennedy
| Participants in “The Evolving Real Estate Valuation Landscape” Appraisal Subcommittee, Federal Financial Institutions Examination Council, 2018 | ||
| Government | Trade Organizations | Industry Participants |
| The Appraisal Foundation (TAF) | American Bankers Association | AVMetrics, LLC |
| Association of Appraiser Regulatory Officials (AARO) | American Society of Appraisers | Bank of America |
| Consumer Financial Protection Bureau (4) | American Society of Farm Managers and Rural Appraisers | Clarocity Valuation Services |
| Federal Deposit Insurance Corporation(3) | Appraisal Institute | ClearBox |
| Federal Housing Finance Agency(4) | Homeownership Preservation Foundation | CoreLogic |
| Federal Reserve Board(5) | Independent Community Bankers of America | Cushman & Wakefield Global Services, Inc. |
| Freddie Mac | Mortgage Bankers Association | Farm Credit Mid-America |
| Internal Revenue Service | National Association of Home Builders | First American Mortgage Solutions |
| National Credit Union Administration | National Association of Realtors | Genworth Financial |
| Office of the Comptroller of the Currency (4) | Real Estate Valuation Advocacy Association (REVAA) | JPMorgan Chase & Company |
| Tennessee Real Estate Appraisers Commission | State appraiser coalitions representative | Old Line Bank |
| Texas Appraiser Licensing and Certification Board | Quicken Loans | |
| U.S. Department of Agriculture | ServiceLink | |
| U.S. Department of Justice (2) | ||
| U.S. Department of the Interior (2) | ||
| U.S. Department of Veterans Affairs | ||
| US. Department of Housing and Urban Development |
For more than 12 years we’ve been testing AVMs and watching them improve over time. More model builders have developed better techniques, and with the falling cost of processing and storage, and with the improving availability of data, AVMs just continue to get better and better.
We aren’t the only ones noticing. We recently read with pleasure Craig Gilbert’s observations of the same phenomenon (Craig is an expert appraiser and co-founder of RAC – Relocation Appraisers and Consultants).
Since co-developing the AVM for Veros in 1999+, I’ve been predicting that AVMs would eventually morph over from Mortgage Origination & Portfolio Valuations, the primary intended uses, into Relocation buyouts. The question has been “when”, not “if”. Relocation represents a microcosm sub-market of the overall residential appraisal business – maybe 5% of the total?
Back in the early days, AVMs were not as accurate as they are today. This has changed. I was thinking about this very thing this morning before opening the current issue of Mobility Magazine, and there it was. The time has arrived.
Read Mobility Magazine December 2018 article “TECHNOLOGY TODAY – What’s Hot for Mobility” written by Steven M. John and Mary-Grace Ellington of HomeServices Relocation.
Here are a few excerpts from the article:
– Recent experiments to test reliability of AVMs show the results to be comparable to formal, in-person appraisals.”
– These valuation tools can save significant time and money while offering convenience.”
– A typical FAVM can be obtained for a fraction of the cost of a traditional appraisal.” [“F” = Forecasting]
– Target values are not fed into the models, and they are not subject to obvious human bias, so theirs perceived impartiality”
– Fidelity Residential Solutions has been at the forefront of testing these new tools.”
Other Resources
Some of you may know Lee Kennedy, an Independent AVM Expert, of AVMetrics, started by Lee in 2005. Lee is a really great guy, has been an appraiser since the mid-80’s, has testified as an expert witness on cases involving use of AVMs and the Financial Crisis and has spoken at recent A.I. Symposium. He’s like the AVM gate-keeper. In his blog titled “The Wild, Wild West of Automated Valuations”, there is a graph showing that the mean absolute error of tested AVMs decreased from 14.7% in 2009 to 5.8% in 2017 and 2018. This is for all AVMs in entire U.S.. Some of course are more accurate than a +-5.7% error rate, when drilling down to specific neighborhoods and AVMs, on a case-by-case basis.
Recently the OCC, FDIC and the Federal Reserve proposed raising the de minimis threshold for residential properties below which appraisals are not required to complete a home loan. Currently, most homes transacting at $250K and above require an appraisal, but Federal regulators propose to raise that level to $400K. A November 30th Wall Street Journal article raises some interesting issues about the topic. They reported that the number of appraisers is down 21% since the housing crisis, but more homes require an appraiser, since more and more homes exceed the threshold each year. The article also states that these factors open the door for cheaper, faster and “largely untested” property valuations based on computer algorithms, also known as Automated Valuation Models (AVMS).
At AVMetrics, we have been continuously testing AVMs for over 15 years, so we’ve seen how they’ve performed over time. As an example, the accompanying chart shows model performance accuracy as measured by mean absolute error, a statistical metric of valuation error. We utilize many statistical measures of evaluating model accuracy and precision, and they all show significant improvement in AVMs over time. And, as these automated tools get better and the workforce of appraisers continues to shrink, the FFIEC members’ proposed change seems warranted, but that doesn’t mean they don’t have their critics.
Ratish Bansal of Appraisal Inc was quoted in The Journal describing the state of AVMs as “a wild, wild West,” inviting, “abuse of all kind.” Furthermore, he contrasts that with the voluminous regulatory standards covering the use of appraisals.
We note much of those voluminous standards represent nearly the same quality control that was in place before the Credit Crisis. In other words, appraisals are not a guarantee against collateral risk. They are simply one tool in the toolbox – an effective, but comparatively time consuming and expensive tool. Also of note, far from being the “wild, wild west,” AVMs are also governed by regulators, most notably, Appendix B of the Appraisal and Evaluation Guidelines (OOC 2010-42) and Model Risk Management guidance (OCC 2011-12). These regulatory guidelines require that AVM developers be qualified, users of AVMs use robust controls, incentives be appropriate, and models be tested regularly and thoroughly with out-of-sample benchmarks. They require documentation of risk assessments and stipulate that a Board of Directors must oversee the use of all models. In other words, if AVMs were the “the wild, wild west” they would be rooted in a town with oversight of the legendary Wyatt Earp.
My strong feeling is that appraisals should not be a sole and exclusive tool when evaluations can be effectively employed in appropriate, lower-risk scenarios. Appraisers are a valuable and limited resource, and they should be employed at (to use appraisal terminology) their highest and best use. Trying to be a “manual AVM” is not the highest and best use of a highly qualified appraiser. Their expertise should be focused on the qualitative aspects of property valuation such as the property condition and market and locational influences. They should also be focused on performing complex valuation assignments in non-homogeneous markets. AVMs do not capture and analyze the qualitative aspects of a property very well, and they still stumble in markets with highly diverse house stock or houses with less quantifiable attributes such as view properties.
However, several companies are developing ways of merging the robust data processing capabilities of an AVM with the qualitative assessment skills of appraisers. Today, these products typically use an AVM at their core and then satisfy additionally required evaluation criteria (physical property condition, market and location influences) with an additional service. For example, a lender can wrap a Property Condition Report (PCR) around the AVM and reconcile that data in support of a lending decision. This type of “Hybrid valuation” is on the track we’re headed down. Many companies have already created these types of products for commercial and proprietary use.
We at AVMetrics believe in using the right tool for the job, and we believe there is a place for automated valuations in prudent lending practices. We think the smarter approach would be to marginally raise the de minimis threshold, but simultaneously to provide additional guidance for considering other aspects of a lending decision, specifically, collateral considerations and eligibility criteria for appraisal exemptions such neighborhood homogeneity, property conformity, market conditions and more.
IAC Opinion Letter to Congress with IAC Listing
On September 13, 2017, AVMetrics participated with its many partners in The Appraisal Foundation in sending an open letter to the House Committee on Banking, Housing and Urban Affairs urging the committee to refrain from weakening the oversight of the Appraisal Subcommittee (ASC). The letter – linked to above – emphasizes the importance of the ASC’s National Registry and the benefits of consistency brought about by ASC’s oversight.
The letter proposes several enhancements to the ASC system, including improvements in background checks for appraisers and a standard definition of a “Federally Related Transaction.” Other proposed enhancements detailed in the letter are designed to improve consistency and transparency.
There is a lot of controversy about appraisals and Appraisers these days, and the FFIEC proposed rule change – increasing the de minimis threshold to $500,000 – allowing for an appraisal exemption and the use of an evaluation in lieu of an appraisal – has sparked anxiety in the world of collateral risk. Our colleagues at the Collateral Risk Network (CRN) expressed their opposition to the proposal. Not surprisingly for a group of its size, there are diverse opinions at the individual membership level of the group. Our opinion is that the change – far from being the catastrophe imagined – will in fact have some important benefits.
While the CRN and certain appraiser blogs expressed skepticism – to put it mildly – we believe that there is a place for an appropriate de minimis level, even the $500,000 level now being considered. On low risk transactions, evaluations (as opposed to full appraisals) can be appropriate and even beneficial for risk management of the overall lending system.
Here’s why. Lending volumes tend to scale up and down faster than the supply of appraisers. As a result, boom cycles in the lending business can place extreme pressure on appraisers. This scenario makes quality control extremely challenging. The option to leverage efficient evaluations on low risk transactions can improve the risk management of the entire system by devoting limited appraisal resources to their highest and best use. In other words, when you place strain on a system, something has to give, and raising the de minimis threshold enables lenders to focus scarce resources on the riskier transactions.
The CRN expressed concern about allowing the mistakes of the recent Credit Crisis to be repeated, and we could not be in more agreement. However, their letter insinuated that evaluations (specifically BPOs and AVMs) were to blame for inflated valuations. Of the vast number and type of quality problems experienced during the credit crisis, evaluations were not a major contributing factor. In fact, we are not aware of any reported cases of AVMs being blamed for the quality problems experienced during the credit crisis.
Strangely, the CRN comments suggested that reviewing individual appraisals is an important source of market trend analysis for investors during overheated markets. We find this highly improbable. The typical single-family appraisal may contain microanalysis of neighborhoods or small markets that lenders may find informative, but most Investors already access market and economic trend data via other sources, including their own or 3rd party economic analyses and risk management tools.
The CRN letter makes the case that appraisals benefit from an extensive regulatory framework and quality control infrastructure surrounding their use, making them inherently safer for the industry to rely upon. We note that much of the same quality control infrastructure and practices were in place before the last crisis. Much of that appraisal quality control depends on the same people and practices – e.g., “desk appraisals” performed by other appraisers – making them subject to similar risk factors. In other words, appraisals are not a guarantee against risk. They are simply one tool in the toolbox – an effective and comparatively expensive tool – but they should not be an exclusive tool when evaluations can be effectively employed in lower-risk scenarios. .
We believe in using the right tool for the job, and we believe that there is a place for evaluations in prudent lending practices. Relying on additional risk measurements, rather than just focusing on a one size fits all de minimis level can provide a formula for better risk management. For example: A $350,000 transaction at a 40% LTV for a pay stub borrower has less need for an appraisal; an evaluation might be able to suffice. Better to allocate that valuable appraisal resource to a $225,000 transaction at 90% LTV. Raising the de minimis, while providing additional guidance for other measures, provides lenders and investors more flexibility to make smarter risk management decisions, and it releases valuable appraisal resources to be used where they can have the most benefit.
Now that the FFIEC has recently closed its commentary period regarding the proposed de minimis lending threshold of $500,000, we expect to receive final communication from the FFIEC during 2016.
We anticipate that lenders will adapt to the new regulations incrementally, with quality controls designed for the new thresholds, not discarded with the bathwater.
Lee Kennedy & Mike Coyne,
AVMetrics, LLC.
