Apurva Mehta, Stanford University

1. Coupling AI with HiTp experiments to
Discover Metallic Glasses
Faster
What to do when theories fail
Apurva Mehta

2. Solutions to many of our current
problems
need
New Materials and Devices

3. New Materials Likely to Exists
30 non-toxic, earth friendly elements  > 4000 ternaries,
> 4 Million compositions : Compositionally Complex Alloys
Metallic glasses
High Entropy Alloys
Etc.

4. Opportunity and Challenge
Rapid Synthesis, Rapid Characterization
But if we synthesize and characterize 1000 alloys (1 ternary
a day)
> 10 years
Just Combi/High Throughput Experimentation Not a Solution
We Need Additional Acceleration/Guidance,
But No Universal Physiochemical Theory Exists,
Plus Processing Plays a Big Role
Over 4000 Ternaries, 4 million
Compositions

5. People who did the work
Fang Ren
Apurva Mehta
SLAC
Logan Ward
Chris Wolverton
Northwestern
Travis Williams
Jason Hattrick-
Simpers
NIST/ U of S.
Carolina
Brenna Gibbons
Bryce Meredig
Kyle Michel
Citrine/Stanford
Kevin Laws, Ron Pandolfi, Robert Tang-Kong, Theodor Ando

6. Boots – on – the - Ground
Machine Learning

7. Outline
• Finding Metallic Glasses, Faster
– New Discoveries of Metallic Glasses
• 7000 new amorphous alloys in at least 10 new
systems
• Challenges (2) for ML for Material Discovery
• From Measurements to Knowledge (labels)
• Combining Knowledge from Diverse Sources
(Gibbons’ Poster)
• Physics from Machine Learning
• Jae Hattrick-Simpers will continue

8. Accelerating Discoveries
New Discovery Paradigm: How to accelerate discoveries when
1. Physiochemical theories fail
2. Processing and other “fuzzy” parameters critical
3. Not a lot of labeled data available

9. Data Driven Discoveries
~6000 reports of
Metallic Glasses
Machine Learning (ML) Model
Landolt-Börnstein (LB) database
In 294 ternaries
How good is this model?
Physics Based Models
Yang’s Delta-
omega model
Efficient-packing
Model (Laws et al)
Legacy Observational Data

10. How do we measure accuracy of a
model?
Perfect Model
False Positive Rate
TruePositiveRate
Receiver Operating Characteristics (ROC) curve
Glass
Not glass

11. How Good is the MG Model?
Perfect Model
Glass
Not glass

12. Processing Condition Dependence
Thermal State and Quenching plays a crucial Role
Melt-spinning Model Sputter Co-deposition ModelVs.

13. Processing Condition Dependence
Different Processing Stabilizes Different Compositions
Prediction of New MG system: Co-V-Zr
Predictions:
Synthesized by Melt-spinning
Predictions:
Synthesized by Sputter Co-deposition

14. High Throughput Validation
Co
V Zr
Parallel
Synthesis
High Throughput
Data Collection
X-rays
2D XRD
Detector
Fluorescence
Detector
Validates One Ternary a Day

15. Challenge 1: Need Labels
Is it a glass?
We measure We need
Image Label

16. Challenge 1: Need Labels
Is it a glass?
We measure We need
Image Label
3

17. Challenge 1: Need Labels
Is it a glass?
We measure We need
Image Label
3
Validation or Leaning only as Good as the Labels
Poorly crystalline
There is a lot of Data, but Labeled Data is harder to find

18. Challenge 1: Need Labels
Is it a glass?
We measure We need
Image Label
3
Can we Automate  Accelerate Labeling?

19. Measurements  Knowledge
Images  Labels
3
Our strategy is:
1. Use Math when we can
2. Capture Expertise
3. Learn/refine
measurements Data Information Label/
Knowledge

20. Automatic Background Subtraction and
Peak Extraction

21. ML vs Experiments
1st Gen ML Predictions
Width Information from Experiments
Experimental Discovery of New Glasses
Predictions from theories

22. Can We Improve Predictions?
Add new data,
Retrain the model

23. Challenge 2: How to add new data to old data?
Add Apples to Apples
Data from LB Handbook
Data from HiTp
Experiments
• Compare data of comparable densities.
• Coverage is still unbalanced.
• LB biased towards positive (glasses)
• HiTp data for only 9 ternaries
Down sampling
Not how much more Data,
But how much new Information

24. 2nd Generation ML Model
Add new data,
Retrain the model

25. Recollect : 1st Gen Model
Perfect Model
Glass
Not glass

26. How much has 2nd Gen Impoved?
Perfect Model
Glass
Not glass

27. 2nd Generation Model  MGs are
Rare
-50
0
50
100
150
200
250
300
350
400
0 500 1000 1500 2000 2500
number of predicted glasses
Less than 5% of Ternaries are glass forming –
less 0.5% compositions are glass forming
Old fashion composition at time (0.5%) expt, and even
combinatorial expt (5%) too expensive without guidance
Experiments
ML Predictions

28. Experiments
ML Predictions
2nd Generation Model  MGs are Rare

29. 2nd Gen Model Predictions: Good Glass Former:
Co-Ti-Zr, and Co-Fe-Zr
ML Predictions High Throughput Experimental Observations
FWHM Glass-forming Label
Discovered Two New MG Systems

30. 2nd Gen Model Predictions: Bad Glass Former:
Fe-Ti-Nb
ML Predictions High Throughput Experimental Observations
FWHM Glass-forming Label
2nd Generation ML model is doing OK,
Biggest problem appears at
the boundaries between glass/no glass

31. In the last year we have found ~
7000 new metallic glasses
in
15 ternaries
(Compare ~ 6,000 in LB in 300)

32. Physiochemical Insights from Machine
Learning
0
0.2
0.4
0.6
0.8
1
1.2
0 20 40 60 80 100 120 140 160
Cumulatefeatureimportance
Work in-progress:
Yang ~8%
Size ~4%
Valence ~5%
Feature number
Current Theories Covers ONLY a few % of the Physics
 New
physics?

33. Physiochemical Insights from Machine
Learning
Jae Hattrick-Simpers – Next Talk……

34. Summary
• Even when 1st Gen ML not exceptional, with
additional data it improves quickly and captures
subtleties hard to capture otherwise.
– Model predicts that < 5% ternary
compositions are glass-formers.
– Accurate Predictions  O of magnitude
acceleration in rate of discoveries
• ML model  seed for physiochemical insights

35. Thanks
35

36. HiTp Measurements  Accelerated Discoveries
Measurements Data Information Discoveries/
Knowledge
a a+b
b
g
d
g+d
Data Quality Assessment
In time for the next measurement
Data Coverage
Assessment
In time for the next
Experiment
AI-driven MGI searches
Knowledge
Base
Robotic Synthesis
“just-in-time”
feedback
On-the-fly
Processing
Unsupervized
Factorization
Semi-
automated
Labeling

37. On-the-fly processing and
visualization of Wide Angle
Scattering data
Sample Raw Data/Images Processed and reduced Data Attribute Extraction and
Real-time Visualization
Calibration  Q-gamma
 1D integration
Signal/noise
Crystallinity
Texture
# Peaks
Ternary
Combi
Library
441 WAXS
Patterns
Ren et al.; ACS Combi,
1
2
3
1
2
3

38. New Metallic
Glasses
Machine Learning
Unsupervised
Information
Extraction
Real Time Data Analytics
 Feedback
1. Stage 1: within time for 1
measurement point
2. Stage 2: time for a data-set
3. Stage 3: time of an
experiment