Video » About NIMH

About NIMH


Watch on YouTube.



Text on screen:

National Institute of Mental Health

The National Institute of Mental Health is the lead federal agency for research on mental disorders.

We aim to transform the understanding and treatment of mental illnesses though basic research and clinical research, paving the way for prevention, recovery, and cure.

National Institute of Mental Health

Original Article

Video » Strategic Plan for Research Overview

Strategic Plan for Research Overview


Watch on YouTube.



Text on screen:

National Institute of Mental Health Strategic Plan for Research

The NIMH Strategic Plan will guide our research forward across the country and around the world.

In this Strategic Plan for Research, NIMH outlines four high-level Goals:

Goal 1: Define the Brain Mechanisms Underlying Complex Behaviors

Goal 2: Examine Mental Illness Trajectories Across the Lifespan

Goal 3: Strive for Prevention and Cures

Goal 4: Strengthen the Public Health Impact of NIMH-Supported Research

These four Goals form a broad roadmap for the Institute’s research priorities beginning with fundamental science and extending through evidence-based services that improve public health outcomes.

National Institute of Mental Health
Strategic Plan for Research

Original Article

Video » Message from the Director on the Strategic Plan for Research

Message from the Director on the Strategic Plan for Research


Watch on YouTube.


Dr. Joshua A. Gordon: At the National Institute of Mental Health, we set the national agenda for mental health research, and it's an exciting time to be at the helm.

Scientific advances are rapidly transforming neuroscience and mental health care.

Computational approaches have revolutionized data science and data sharing.

Every day, we're learning more and more about the brain, and gaining a better understanding into the origins and causes of mental illnesses.

We've developed the NIMH Strategic Plan for Research to take advantage of these opportunities.

Using this plan as a guide, NIMH aims to support high-quality, impactful research and to translate this research into clinical practice, services delivery, and policy. Learn more about this plan at

Original Article

Video » Zhen Ni: Transcranial Magnetic Stimulation: Cortical Anatomy and Clinical Neurophysiology

Zhen Ni: Transcranial Magnetic Stimulation: Cortical Anatomy and Clinical Neurophysiology


Watch on YouTube.


Zhen Ni: So, my name is Zhen Ni. I am a Senior Research Fellow working in NINDS. So, today I will talk about the Cortical Anatomy and the Clinical Neurophysiology for the TMS Introduction. As we know, TMS, the Transcranial Magnetic Stimulation, is a powerful but non-invasive technique to stimulate the human brain. And the — it's a very good technique to study the human brain function.

So, as we can see from this slide that if we give a stimulation to the human brain, it produces a descending volley in the spinal cord. And then, you can activate the spinal motor neuron pool. In transmission, you can activate to the target muscle then we can record the response in the target muscle. So, in this talk, I want to focus on four different topics. The first one, how can we record the response from the — from the stimulation of TMS? Second, how can we test the spinal cord activity? The third one, how do we do the motor cortical stimulation with TMS? And what is the descending volley of the TMS? And the fourth one, I want to focus on the stimulation outside of motor cortex.

So, let's go to the first part. How do we record the response of TMS? There are — usually, there are two different techniques. One is electroencephalography, EEG, and there's another technique named electromyography, EMG. EEG is a very good technique to record the activity of the brain. So, it's usually recorded using a system named the 10-20 EEG System. Here, 10-20 refers to the factor that the actual distance between the adjacent electrodes is either 10 percent or 20 percent of the total front to back, or left to right, distance of the skull.

Here, we measure the distance between the left ear and to the right ear. Also, we measure the distance from Nasion point to the Inion point as the total difference. We can mount many electrodes on the scalp. Here, we use the abbreviation of F as the frontal area. And the temporal area can be shortened by the T. Parietal area is P, and the occipital area is O, and the central area is C.

So, there are many different techniques to record the EEG. We can use a cap, for example, like showing in this slide. We use a 64-channel cap to do the EEG recording. So, there is a very important step to record the EEG. That's the preparation of the cap or electrode in other techniques. The preparation — the purpose of the preparation of the cap is to reduce the impedance of the electrodes. And in this slide, the red color means it's a very high impedance. And the green color means low — lower impedance, which we can use it for the next step for the recordings.

So, like in this slide, it's a new cap; it's named active cap. On the left side, it's a cap before the — before the preparation. And we can see the technician is using the gel to prepare for the cap. And on the right side, almost all lights turn to green. That means the cap is ready for the recording. And in the middle, that means we are doing the preparation.

There's another technique named the electromyography, EMG. This can also be used to record the response to TMS. And the mechanism behind the EMG is a size principle. The size principle means there are many motor neurons in the spinal cord. And these neurons can be recruited in order of — from the smallest to the largest of size when the muscle is doing contraction, and the contraction is an increase little by little. So EMG, with different techniques, can record the activity of muscle fibers. And the EMG recorder can reflect the excitability of the neurons connecting to the muscle fibers.

In this slide, we show the recording of a single motor unit. Here is a very important concept named the single motor unit. That means a single motor unit includes an alpha motor neuron and all the corresponding muscle fibers it innervates. That means all the fibers which connect to the motor — connect to the single motor unit — have the same firing property. But when we use a needle to insert into the muscle, and the tip of the needle is close to the muscle fibers, we want to record. At that moment, we can get to the discharge of the muscle fibers. And this firing property of the muscle fiber can reflect the firing property of the motor neuron, which it — which is connected to all these muscle fibers.

So, on the right side, we can see an example recorded of the single motor unit recorded in the — in the first dorsal interosseous muscle. So, there is a very clear limitation of single motor unit recording. At first, of course, it's invasive and painful. And another important limitation is that it's almost impossible to record all the muscle fibers in the muscle with the single motor unit recording.

So, over to — to overcome this limitation, we can use the surface EMG. So, see surface EMG is used to monitor the general picture of the muscle activity. So, the surface EMG can superimpose the activation potential of the muscle fibers under the electrode we use. So, here we used an electrode attached to the surface of the muscle. So, here is the concept named the compound muscle action potential, C-M-A-P, CMAP. It's a very useful measurement. And it can be recorded with surface EMG. And the CMAP represents the summation of almost simultaneous action potential from many muscle fibers in the same area. And usually, CMAP can be — can be evoked by stimulation of the motor nerve.

So, with this surface or the single motor unit recording, we can test the spinal cord activity. The next part I will talk about, the test of spinal activity. So, in the clinical neurophysiology, a stimulation is often used to demonstrate whether the stimulated structure. For example, brain, spinal cord, this structure is involved in the — in the special movement task. And we can analyze the response to the stimulation to look at the effect. So, spinal activity can be tested by stimulation on the afferent pathway of spinal motoneurons.

Here, we should say the basic mechanism, which underlining this test, is that the corticospinal neurons from the brain in the motor cortex send their fairly long axons down to the spinal cord and the synapse on these spinal alpha motoneurons. And the alpha motoneurons send the fibers, which can synapse on the muscle fibers. And this is the physiology called — or — and the anatomical mechanism for testing the spinal cord activity.

In this slide, we introduce the monosynaptic reflex in — where — which is related to the spinal cord activity. The very classical monosynaptic reflex is a stretch reflex. So, we can see on — in the picture when the doctor taps the tendon of the muscle, the muscle spindle can be activated. And this will induce monosynaptic reflex, and we call them tendon reflex.

On the right side is the pathway for the tendon reflex. We can see the muscle spindle can be activated by the stimulation through a 1a afferent. It goes up to the alpha motoneuron with a transfer in the alpha motoneuron it goes — it sends the pulse to the muscle through the efferent nerve, which is the motor nerve. On the physiological level, we can do a technique named the Hoffmann reflex, H-reflex. And the H-reflex is the analog to the tendon reflex.

On the left side, I show the example recordings for the H-reflex. I should mention here that when we do H-reflex, at the same time, we can record the muscle wave, M-wave. And this M-wave is similar to the CMAP, which I showed in the previous slides. So, in the recordings from bottom to the top means an increase of intensity. At the very bottom, when we use various low intensity, no response can be recorded. When we increase the intensity, H-reflex appeared first. When the intensity was increased again, the H-reflex become larger. And, at the same, M-wave appears.

And when the intensity becomes very high, the H-reflex becomes smaller and smaller while the M-wave becomes bigger and bigger. When the stimulus intensity reached a very high level, H-reflex disappeared while M-wave reaches its maximum. So, the mechanism behind the — this H-reflex that the stimulus comes — the stimulation, the electrical stimulation, given to the nerve can go both to the afferent — go to the afferent and the efferent fibers in both directions.

So, in the — in this slide, I show the H-reflex mechanism. On the left side is results. The group analysis results for the recordings shown in the previous video. So, the X-axis means the stimulus intensity, and Y-axis means the amplitude of H-reflex and the M-wave. We can see that the H-reflex has lower firing — a lower threshold. And when the stimulus intensity increases, it reached to the maximum. And then, the decrease is the increase of the stimulus intensity.

On the other hand, the M-wave has lower — has a higher threshold, and the increase almost linearly with the increment of stimulus intensity, and then finally reach the maximum. On the right side, is the mechanism behind the H-reflex and M-wave. We can see the red line means — red arrow means the stimulus given to the motor nerve, and the blue arrow means stimulus given to the sensory nerve. With the single stimulation, the inputs can go both ways. The antidromic nerve way and the orthodramic way on both sensory and the motor nerve.

So, the orthodramic impasse on the motor pathway can produce an M-wave. And the antidromic pathway — antidromic impasse on the sensory pathway can produce an H-reflex. At the same time, the antidromic impasse on the motor nerve can cancel the H-reflex. So, that's why we can get to the experiment results, which I discussed previously.

So, with all these slides, we can know the H-reflex is analog to the stretch reflex in the physiological lab. And the H-reflex is often used to test the spinal activity in motor physiology. That means if we use a TMS to stimulate the brain and get a response in the muscle if the — we ask the subject to do two different tasks and we found a different response. At the same time, we must think about the spinal cord activity as the spinal cord is on the pathway of — from the motor cortex to the — to the muscle, which we want to record.

So, there's a limitation for H-reflex. That is, the H-reflex is sometimes very difficult to be recorded from hand muscles. So, as a replacement, we can do an F-wave experiment. The mechanism behind the F-wave is that when we use very, very high stimulus intensity, the H-reflex can completely be blocked by the antidromic current, as we discussed before. At that time, no H-reflex can be recorded.

But, at the same time, the antidromic current on the motor fibers is this; we call it super maximum intensity. Such very high stimulus intensity can activate the motor neuron directly, and they induce a wave. This is named the F-wave. So, there is an implication for F-wave. We also can use it — use F-wave to test for the spinal cord activity. So, there's very clear limitation for F-wave that the F-wave only reflects the excitability of motor neurons with very high firing threshold.

There's an immediate and very important implication for the F-wave recording. This is named the central motor conduction time, CMCT. So, when we give a stimulation to the cortex we can get, we call them motor evoked potential, MEP, in the muscle. When we measure the MEP, we can get to the latency of this MEP. So, the MEP latency included two parts; one uses a central motor conduction time, another is a peripheral nerve conduction time.

Let's go back to the previous study, use F-wave. When we give electrical stimulation to the motor nerve, it goes up to the spinal cord and then goes — then comes down to the muscle. This produces latency. If we add a slight latency with the M-wave latency, we can get two times of the peripheral nerve. So, at this time, we should consider a synaptic delay on the transfer from the motor nerve to the spinal cord and then come down to the muscle. So, in the form — in the formula, we should subtract one synaptic delay, which is a one millisecond.

So, finally, the formula comes up like the central conduction time equals to the MEP latency minus F-wave latency plus M-wave latency minus one divided by two. So, the central conduction time can also be measured with TMS alone. In this figure, we can see when we measure the conduction time in the hand muscle, APB muscle on the thumb, we can give a stimulation to C3; that's close to the motor cortex, and it produces MEP.

And if we give the stimulation to the cervical level, it produced MEP with shorter latency. If we get to the difference between two — between two MEP at a different side we can get — finally get to the central conduction — central motor conduction time. A very similar technique can be used to measure the conduction time in the leg muscle, tibialis anterior.

Then, we come to the next question. How can we do a motor cortical stimulation with TMS? This slide shows the primary motor cortex. The primary motor cortex is identified by Brodmann area 4. So, the primary motor cortex contains very large pyramidal cells, which is a cortical spinal neuron. These neurons send a very long axons down to the spinal cord and the synapses motoneuron pool. And the important primary motor cortex is a target for many TMS studies.

The physiological and the anatomical mechanism behind the TMS targeting motor cortex is showing in — is shown in this slide. It's named the cortical homunculus. The cortical homunculus was founded by a Canadian neuron — neurosurgeon named Penfield. And Dr. Penfield used a needle — used a needle stimulation when he did neurosurgery, and he stimulated the motor cortex and created a map, which is shown in this slide.

So, what we can see is that the presentation of different muscles in the — in the primary motor cortex are highly disproportionate, and this is named the cortical homunculus. With this mechanism, we — that's why we do many TMS studies using muscle — we use very small hand muscles, as the small hand muscles have a large cortical presentation and can be easily recorded with surface EMG.

This slide shows equipment for TMS — for TMS study. In the TMS lab, there are several different devices that should be included. The first one, of course, is a stimulator of TMS. And this stimulator can be connected to the — a coil, and the coil delivers the stimulation to the brain. And with this stimulation, we can record the — from the small hand muscle. For example, the first dorsal interosseous muscle, the FDI muscle.

For many studies, we use different combinations. For example, we do a single pass; at that time, we only need one stimulator. And sometimes, we need two stimulators to do a paired-pulse protocol. At that time, we need a bistim to connect two different stimulators together. This bistim will show on the top in the — in the figure. And when we record the muscle response, we should use the amplifier, as the amplitude of the response is very small. The response, as shown on the right side, it's named the motor evoked potential, MEP.

So, this slide shows the electrical mechanism of the transcranial magnetic stimulation. When we give this stimulation, it — the stimulation produces a very large but brief current in the wire coil. And this brief but large current was produced by discharging of a bank of capacitors, which can be seen in the previous slide. After — when we give the stimulation to the brain, there's a secondary induced current in the brain. The current induced in the brain has the opposite direction to the — to the current in the coil. Here we should mention that in the figure, this — the magnetic field is produced by a round coil. There are many different coils with different shapes. And the round coil is not now — not well used in many, many studies.

So, this slide shows the magnetic coil with different shapes. The first A coil is the same one showing in the — in the previous slide. It has a weak — it produced a weak stimulation, which can — we can see the electrical field on the right side. Then, this idea to improve the coil shape with a different purpose. B coil and the C coil focus on the tip, the electrical field focus on the tip of the coil. And the D coil has a large coil with high stimulus intensity. And the E coil is small in size, which produced focal stimulation.

And the many studies use the figure of 8 coil, the F one. The figure of 8 coil used two coils, which produce this current with the same direction at the joint point of two coils. So, that means at the center of the — of the figure 8 coil, the electrical field is the strongest. The figure of 8 coil is now well used, and the most used in many TMS studies.

This slide shows the example recording with a single stimulation to a single site, but with a recording in different muscles. On the top is an electrical stimulation, and the bottom is a magnetic stimulation. In the early years, electrical stimulation was used to stimulate the brain. So, the disadvantage for the electrical stimulation is that when the current goes through the scalp, it activates the pain receptor in the scalp, and that's why the electrical stimulation is very painful, and that's why electrical stimulation cannot be widely used. And later, people developed the magnetic stimulation.

In this figure, we can focus on the motor evoked potential latency. What we can see is that with the stimulation — with the single stimulation, the MEP latency is the shortest in the biceps muscle and they become longer in thrombus — in the thumb, which is recorded in APB muscle. And it's longest in the leg muscle, tibialis anterior. The latency becomes longer and longer from the upper limb to the low limb is consistent with the anatomic findings, anatomical location of different muscles. The second evidence, which can be found in the figure, is that the magnetic stimulation produced MEP with a slightly longer latency than the electrical stimulation, which we will discuss later.

And this slide shows the stimulation with different — at different locations, but recording in different — in the same muscle. The recording was made in the APB muscle. The top one shows a recording right hand, and the bottom one shows recording in the left hand. What we can see is that the first trial on the very top is recording from electrical stimulation at the wrist, which is similar to the M-wave we discussed before.

The second trial is recording from TMS at the cervical level. It produced a little longer latencies than the M-wave. And the third line shows that MEP with TMS at the cortical level. It produced the longest MEP latency in three trials. This is also consistent with the anatomic locations of different stimulation. Here, we discuss the site of stimulation. The bottom one is the electrical stimulation to the motor cortex. Here, the cathode electrode is located in the center of the skull; we call it a vertex. And the anodal stimulation is located in the motor cortex, close to C3, C4.

So, the current produced by this electrical stimulation is a vertical current. It goes down from the surface of the cortex to the very deep area in layer — in layer five, six where the large cortical spinal neuron is located. It is a little different for the magnetic stimulation on the top. The magnetic stimulation, which we discussed previously, it's parallel to the surface of the cortex. So, the stimulation only goes to — only goes to the inter neurons in the layers two and the layer three. And with both stimulation, the stimulation — with both stimulation of the electrical stimulation, and the magnetic stimulation the stimulation is given to the primary motor cortex, which can see from the right slide. The primary motor cortex is located at the anterior bank of the central gyrus or central sulcus.

So, we discussed that the magnetic stimulation and the electrical stimulation are different. As the magnetic stimulation is a parallel to the motor cortex, which activated the layer two and the layer three neurons in the motor cortex. And the electrical stimulation can activate the pyramidal cells located in the layer five and layer six. This led to a very important concept in TMS, which is D and I-waves. D-wave, which is a direct wave, it shows a direct activation of the pyramidal neurons. And I-wave is an indirect wave. It's indirect wave reflect the indirect activation of the cortical spinal neuron through the synaptic mechanism.

In this figure, we show the cortical spinal wave, the descending wave recordings with implanted spinal cord electrodes. On the right side, is an x-ray photo for the electrode. We can see the electrode. The spinal electrode was implanted at the cervical level. There are four contacts, zero, one, two, three on the electrode. When we give stimulation to the motor cortex, the impasse goes down to the spinal cord. Then, we can record a very tiny potential named the descending wave from contact to zero and the three.

These recordings are shown on the left side. And the — on the middle is the MEP recording with EMG electrodes. There is a red line in each — in each recording figure. For the cortical spinal wave recording, the red line means the D-wave latency. And then — on the right side, for the MEP recording, the red line means the T — the electrical stimulation induced the T — the MEP latency.

There are different panels. On the top panel, is the recording for electrical stimulation. What we can see is that with the electro stimulation, it produced a D-wave. And the second line is the TMS with a lateral, medial current. With this current direction, we also can produce D-wave. But following this D-wave, this — the indirect wave, I-waves, the bottom electrodes are — the recordings with posterior-anterior TMS stimulation. With this direct current stimulation, we can see there is no D-wave can be recorded. The first wave appears as the I-wave, I1-wave. Then, there's a series of different waves we name it I1, I2, I3.

So, from this recording, we can get information that the lateral medial direct current first generated D-wave. And the posterior-anterior current generates an I1-wave, and the more waves can be generated when stimulus intensity increases. So, different waves can also be recorded with current or with TMS with different currents. In this figure, at the top is the recording for TMS with latero-medial current. As we already discussed, a D-wave can be generated, followed by later I1, I2, I3 waves.

And in the middle is the stimulation with the posterior-anterior current, which can be — which can be seen in most of TMS studies. With this current direction, no D-wave can be recorded. Only I1-wave can be recorded with low intensity by incrementing stimulus intensity produce more waves, including I2 and I3 waves. And at the bottom, the anterior-posterior current initially produced I3-wave with low stimulus intensity. And with incrementing stimulus intensity, more waves, including I1-wave, can be seen.

And this slide shows the recording with a single motor unit. This — the experiment was also done for recording different waves with different stimulus current. On the recording on the left side is the MEP recording. And there's a dashed line showing the latency with LM current direction. On the right side, is the single motor unit recording with the analysis named the Peristimulus time histogram, PSTH, analysis.

So, on the top of each panel is the recording with the posterior-anterior current. What we can see is that the MEP latency is slightly longer than — with slightly longer than the dashed line, which shows the MEP latency with the lateral, medial current. And with the single motor unit recording, we can see there are three different waves that can be recorded I1, I2, and I3 waves. On the other side, the anterior-posterior current, which is showing in the bottom, with this current MEP latency is longer than the posterior-anterior current. And with the single motor unit recording, we can see that only later I-waves, including I3-waves and the I4-waves, were recorded.

This slide shows animal studies, which discusses the mechanism of later I-waves. On the left side is the stimulation to the motor cortex. And the recording is also the spinal cord recording in a monkey. With the motor cortical stimulation, we can see a series of I-waves can be recorded. At the bottom, with a stimulation to the pre-motor cortex, no I-waves can be recorded except for the stimulation with very high intensity.

So, on the right side showing the two stimulation was — were given along — were given together. What we can see is that when we give pre-motor stimulation with the motor cortical stimulation, the later I-waves were facilitated large — were largely facilitated in both monkeys, CS-14 and the CS-17. That means the pre-motor cortex is not directly producing the later I-waves, but is involved in the production of later I-waves.

This figure shows the leading hypothesis in the field, which can explain the production of cortical spinal waves. With a single stimulation of TMS, many neurons can be activated in the primary motor cortex; these neurons, including the larger cortical spinal neurons in layer five, we call them P5. At the same time, the facilitatory inter neurons in layer two and the layer three, we call them P2 P3, can be activated. Also, there are inhibitory inter neurons in the motor cortex. Some of them are mediated by GABA transmitter, which is shown in the black square.

So, with a single stimulation, all these neurons can be activated. The activation of the axon of P5 can produce the D-wave. And the activation of the P2 and the P3 neuron can produce the I1-wave. So, when the stimulation activated the GABA neurons, it can eliminate the activation of I1-wave. This is the inhibitory phase of the I1-wave after the activation of I1-wave.

So, when the P2 and the P3 are activated again, and the GABA neuron stops firing it — the later I-waves are produced. When we use a different current direction, for example, the anterior-posterior current direction, we may activate all these neurons. And the — these — all these neurons can be influenced by the inputs from the pre-motor cortex. And this is shown in the larger circle around all these areas. So, this figure shows the leading hypothesis in the field. But there are many other models that can explain the mechanism behind the very complex cortical spinal descending volley. And all these models and the hypothesis should be tested in further experiments.

So, let's move to the fourth part of the talk. TMS can also be applied to different cortical areas outside of the motor cortex. This one is showing an important TMS technique named the TMS Mapping. The coil of — the TMS coil is moved in a different direction. On the X-axis, it's moved from the medial to the lateral side. And the Y-axis showing the coil moving from the posterior to the anterior locations. At the center of this figure, showing the location which produced the largest MEP amplitude, we call it the center of gravity. And when TMS coil is moving around this — around this point of the center of gravity, the MEP size becomes smaller.

And finally, we can get an MEP map from this technique. This technique has many different but very important implications. For example, in the patient with amputation, the recording on the intact side was shown on the left, and the amputated side was shown on the right side. And the recording was made from biceps — and the biceps muscle. And the patient has upper limb amputation, but the biceps muscle has remained. What we can see is that the amputated side has a much larger activation area than the intact side, which reflects the more activated state after the amputation.

The MEP mapping technique can also be used for the motor learning experiment. In this experiment, the subject does a serial reaction time task. That means the people — the subject used four fingers from the index, middle finger, ring finger, to the small finger, the little finger, which we’re referring to as one, two, three, four. And the subject was asked to do a serial reaction time task with a different sequence, which can be shown on the screen. And this did a very long time to learn for the subject. And the learning of the course can be recorded. And the performance during the learning as shown on the right side, we can see both the reaction time and the performance become better after many, many blocks.

On the left side is the brain mapping, the MEP mapping, for each different block. We can see, before the training, the stimulation didn't cause any activation of the corresponding muscles. But with the stimulation — with — when the training course is going on, the activation of the involved muscles become larger and larger. And finally, at the block nine, the TMS map become the largest. So, that means the cortical areas for muscle involved in the task increased after the motor learning process.

TMS can also be given to other different cortical areas, as the — there are very complex cortical networks in the brain. And the TMS can be used to test the connectivity between different cortical areas. In this figure, we use the electroencephalography, the EEG, to record the effect of TMS. And if we analyze the different response, the different component, and the different location for the TMS induced effect, we can know TMS at one side can activate the — which area and which component. TMS can also be given to a different cortical area.

In this figure, we show the — show the recording with electroencephalography after TMS on one side. And with analysis with different components and at different locations, we can know the TMS on one side can activate the remote cortical areas. But with EEG recording, there's an important limitation to do the experiment. That is, the TMS evoked potential, we call it TEP, the TEP may be technically difficult for the very large artifact produced by TMS. So, one way to overcome this problem is to select the TMS compatible caps in the experiment. We can also use different software to remove the TMS artifact with such kind of experiment. But with all these efforts, the TMS artifact is still a very big problem in the area of the TMS evoked potential.

The TMS can also be given to the non-motor cortical areas. Sometimes this can lead to very high impact studies. For example, in this nature experiment, TMS was given to different cortical areas. On the left side, we can see the experiment was set up. In this experiment, the authors compare the effect of TMS in the very early blind subjects to the healthy controls. So, all the subjects performed a Braille — a Braille reading task, which shows on the left side. At the beginning of the task, the subject moved their index finger to start the Braille reading task. When the finger moved to the left side, we can see there is a laser beam on that. When the finger covered the laser beam, it triggered a TMS trait with three seconds at one — at about 10 hertz. With this train of stimulation, it produced a virtual lesion in the stimulated area. The results are on the right side.

For this braille-reading task, the subject should read out what the characters they have read during the stimulation. And what was measured is the error of the reading task. The interesting thing is that, in the healthy control subject, the stimulation on the sensory motor cortex produced a lot of error in the subject. This is because the sensory inputs were impaired by the train of stimulation. Another very important result is that, in the blind subject, the sensory motor stimulation did not impair the results. But the middle occipital stimulation was — which is targeted on the primary visual cortex impair the — impair the task, and they impair the performance of the subject a lot. So, the results suggest that the visual cortex is involved in the cognitive task in the early blind subjects.

This is the follow-up study for the previous one. In this experiment, the stimulation — the set up was changed a little bit. The motor task is that the subject was given — was presented a word with auditory stimulation. For example, in this subject, the apple — the word apple was presented to the subject. And with this auditory stimulation, the subject should respond with a logically correct word, for example, eat. But the subject cannot respond like a drink or other words.

At the same time, a train of stimulation at 10 hertz lasted for three seconds was given to different cortical areas. These areas, including the visual cortex, the prefrontal cortex, the somatosensory cortex, and the lateral occipital cortex. The interesting thing is that, in the site control, in the healthy control, the prefrontal cortex stimulation impaired the performance of the subject. That means in healthy control, the prefrontal cortex is important for the cognitive tasks.

But the result says different in the subject with early blind. Only visual cortex — visual cortical stimulation produced the impairment of the motor performance. So, all these experiments show that the stimulation, the TMS stimulation, to different cortical areas can be important for the — for the cognitive and the motor learning tasks.

Going to the summary of this lecture, at first, TMS is a powerful neurophysiological technique to study human brain functions. The effect of TMS can be recorded with electroencephalography and the electromyography. Spinal activity should be taken into consideration for a TMS study. Motor cortical stimulation activates the cortical spinal neurons, and it produces descending cortical spinal waves. The effect of TMS at other cortical areas are often complex, and they may interfere with motor learning and the cognitive process.

So, again, I am Dr. Zhen Ni. I am working at the National Institute of Neurology Disorders and Stroke. I am a Senior Research Fellow. Thank you very much for watching this video. I hope this video can help you when you do TMS studies. Thank you very much.

[end of transcript]

Original Article

Video » Bruce Luber: TMS Parameters and Protocols

Bruce Luber: TMS Parameters and Protocols


Watch on YouTube.


Bruce Luber: Hello, my name is Bruce Luber. I'm a staff scientist at the NIMH, National Institute of Mental Health, and I'd like to talk about some of the basics behind TMS, transcranial magnetic stimulation, by talking about the basic parameters – the physical parameters – that we use to change TMS as we apply it to the brain, and then give you some idea of some of the protocols we use to test its effects.

So, what is TMS? Basically, we're looking at — by B field, I mean magnetic field — we're looking at magnetic field, the changing magnetic field, which produces a changing electric field inside the brain, and that leads to depolarization of neurons in the brain locally, right near the coil that we use to produce it. So, there's, for instance, an example of a coil. The magnetic field surrounds the coil and sort of a toroid shape in this case. As you see, this yellow line moving around, that's the electric current moving around the coil. Inside of it is a very large current. It creates a magnetic field that's on the order of an MRI machine, for instance. It can go up to two Tesla. And that — and the thing is, that that electric field only lasts for less than a millisecond. One pulse is less than a millisecond long. It's a very fast rising and falling, changing current that creates a magnetic field, also very fast rising and falling, which induces electric current in the head near the coil, and that creates changes — this is a little section of an axon — changes in a neural membrane such that you get neuronal depolarization. And that can lead to subthreshold changes even before an action potential gets created. So, if the field's not big enough to actually create an action potential, it still can create effects in the membranes of the cells and the synapses of the cells. That can create longer-term effects in terms of intercortical inhibition or intercortical facilitation or excitation. But it also does create — when it's a strong enough field — it will also create action potentials that set the neurons off. And that, of course, leads to neurotransmitter release in the synapse of the cell, and that further leads to synaptic action there and then sets off the neurons that it's connected to and so on in the brain leading to changes in behavior.

So, I'm very excited. I'm an experimental psychologist. I'm very excited about TMS myself. I've been doing it for 24 years now because you can non-invasively affect behavior, change the brain, and test what's going on where locally in the brain for various functions like memory or sensation. And — but it's also very useful in terms of therapeutic applications. So, TMS is already FDA approved for depression and OCD. And it's already being looked at in terms of long-term changes it might be able to make in schizophrenia and anxiety. It's been used for a long time looking at pain and looking at recovery from stroke as well as movement disorders. And it's also looked at and studied for, in terms of tinnitus and epilepsy. And it's basically being used in any kind of thing that neurologically or in terms of psychiatry can- is substantiated in the brain and can we do things about it. There are many targets of TMS in the brain, the dorsal lateral prefrontal cortex, the temporal parietal areas in the brain, motor areas, and motor systems in the brain, temporal memory systems, and sensory systems as well. So, this is a tool that can be used in a variety of ways, and hopefully, in many therapeutic ways. For instance, TMS, like I said, has already been approved by the FDA for use in depression. And this is an example of a — one of the large studies that lead to that approval. And you can see the blue columns — the blue columns that you see here, show that the rate of — a response rate in terms of depression increasing as you go from week two to week four to week six, okay. And we actually give TMS — the way that they give TMS in the world right now is to give it for six weeks. You can see that six weeks — after six weeks- there was a notable increase of response to the TMS and a decrease in depression relative to placebo. So, that's why we're using TMS.

But let me just start talking about the various parameters that are involved in TMS. There are a great number of them. So, there's a single pulse that's given. And so, each pulse has certain characteristics to it. And you can have a certain shape to it. The shape that you actually use determines how the neuron is locally — the membrane- is locally affected by it. But you can also do more than one TMS pulse. You can do trains of TMS pulses, and that provides a number of other parameters. So, you certainly can talk about the intensity of the pulse, all right, how strong we make the pulse. But also then, in a train, the frequency of the train, how many pulses per second you apply, how long the train is, how much time between trains you allow, how many trains you do, and then how many sessions of all these trains that you do. All these things can be explored. And it even gets more complicated because you don't have to have one frequency. You can have one frequency inside of another frequency.

Another FDA approved form of TMS now is called theta burst. And in theta burst, you actually do small bursts of pulses that are high frequency, say 50 times a second or 50 hertz. And then do those bursts every 200 milliseconds. And that means you're doing them five times a second or at five hertz. All right, so you're doing five hertz and 50 hertz at once. It's called theta burst because theta is like — the theta rhythm in the brain is about five hertz. Fifty hertz is actually the intrinsic gamma rhythm in the brain. And so, you're using some of the rhythms in the brain and nesting frequencies inside of other frequencies in a biological way that's actually interesting and probably very potent in the brain. This is called patterned stimulation. And it's just another way to combine intertrain intervals and frequencies in the brain. And these are just the temporal parameters. There's also spatial parameters. So, the shape of the coil that you're using. The target, the actual spatial target — where in the brain you stimulate are also very, very important to what happens with the TMS.

So, I'd like to go over some of these parameters, and I'll start with the spatial parameters themselves and basically looking at it in terms of targeting our coils. So, and I — there are a number of different ways to target your coil outside the head to aim at different places in your brain. Now, you can do it — the FDA approved a method that originally — I'm sorry, the originally approved method was to use a probabilistic location. In other words, to find the place in the brain and over the motor cortex that produced a stimulation of a muscle that you could see, for instance, a hand muscle that you could see and then just move five centimeters in front of that. And that placed you approximately over the dorsal lateral prefrontal cortex, which is a target for alleviating depression.

Now, another way of doing scalp positioning of the coil is to use the 10-20 — the International 10-20 EEG system that sets up exactly where, in a relative way depending on the size of the person's head, you put electrodes on the brain. And there are electrodes that are right over — that we know from imaging- are right over various places like dorsal lateral prefrontal cortex in the brain. So, you can use that system as well to position your coil. These aren't great ways to do it because you're essentially blind to what's the actual location of the brain you're looking for. So, you can use things like MRI, say a structural MRI, of that person, and then be able to target based on the person's MRI. Since everybody's a little bit different, their brains are formed a little bit different, where everything winds up in the brain is a little bit different in each person. This allows you to target much more precisely the site that you're looking for to stimulate. You can do very nice targeting on the scalp in one sense. You can look for a spot, say over the motor cortex, that will stimulate a certain muscle, okay. One muscle rather than another muscle, just by moving the coil around and stimulating and watching the muscles move.

So, that's one of the very few times you can do this on the scalp in any kind of precise way. You can also do it in terms of phosphenes in terms of the visual system. You can create phosphenes in the brain by stimulating the back of the head over occipital cortex where the visual cortex is. And you can move it around and find the best spot to stimulate there. And you can do that actually in different places in the visual field. So, you can do it relatively precisely there too. The rest of the cortex is basically silent to that kind of procedure. So, we need imaging to really help guide us if we really want to precisely aim our magnetic field.

And so, other ways to do it are by looking at a structural MRI and targeting that way. And further, you can actually use functional MRI. You can have people, for instance, do a task — say a memory task — look what lights up in the brain, and target that spot that lit up in the brain as well. That's a little bit more precise in terms of being able to stimulate the part of the brain that is functional and associated with the function that you're trying to change in the brain. You can take that one step further because you can do that on a group basis, which is kind of how you'll see most MRI research. But you can do this on an individual basis, which is really where we'd like to go in terms of helping people and doing — finding therapies. Because each person's different in exactly where a certain function might be instantiated. And so, you can actually do imaging and do fMRI on an individual basis and look at where in the brain that person is using the brain for that given function and aim at that. And we think that's probably the most precise way to do this.

So, just in terms of EEG sites, you can see, for instance, that these are the kind of the general regions: this is based on a number of people that were brought into a CAT scan with the electrodes on they saw what part of the brain they are actually- a given electrode site actually winds up being over. And there's a kind of a large multi-centimeter area that the electrode could be over in any one person, what cortex it's over. The important one for depression, for instance, is — would be the site over the dorsal lateral prefrontal cortex. So, if you use the site F3 in the EEG 10-20 system, you may wind up somewhere — and I just lost it — there it is — somewhere over here. So, it's not quite precise, you know. It's not a very specific region. But, at least it gets you over dorsal lateral prefrontal cortex, and that's kind of the best you can do with scalp-derived targeting at the moment.

You can use what's called neuronavigation. You can use infrared cameras that can very, very precisely pinpoint exactly where the coil is relative to an MRI online with a person. So, you can have a coil be over a person's head and know exactly where you are. And even if a person is moving a little bit, you can see exactly where the coil is relative to the target in the brain. So, this is something that we all use now: this kind of frameless stereotaxy.

Just in terms of a comparison of these different methods: there was a group, Sack et al., in 2009, that ran a study to compare these. And I won't go over in detail the tasks they used. But they found a task, a Stroop-based task, and basically, they showed you two numbers. And the numbers could be the same size. That's the neutral condition. But they could also show you two where one was bigger than the other. Now, the way to answer this when these numbers come up, you say which one is bigger. And they mean bigger in terms of — like four is bigger than two. But in this case, it's actually also in size; it's bigger as well. And so that actually helps you answer faster. Whereas if the two is bigger, you kind of get slower because, in one sense, two is bigger, in another sense, four is bigger. And that slows you down a little bit. And you can just see that over here. If this is the reaction time, you can see that when there is no TMS that you answered about this many — about 480 milliseconds, you're a little faster in the congruent case when the four, for instance, is bigger. And you're much slower in the case where they're incongruent.

So, you have this normal effect without TMS in general. The TMS itself, when you stimulate the right parietal area, you actually get faster. It makes you faster in the incongruent stage — condition, rather. And so, this is a TMS effect that enhances your performance slightly. And we can look at how many people it takes to see this kind of effect — how many people it takes to statistically see how many it takes using the various targeting methods. And so, this was done. And so, they used the scalp 10-20 system. So, they went over right parietal cortex based on the scalp. And when they did, that's this group of five people here — five brains — you can see that they stimulated in different places in the brain when they actually looked at their MRIs. So, there was a variation in where they ended up over the brain if you just used a scalp-based system. That variation is just random in the sense of what's going on in the brain functionally. They used MRI-guided — just used their structural MRIs. Then you're going for a specific spot in the brain that they found in a group level activation. And the other way they tried to do this was to do the individual guided MRI.

So, they found in each of these brains where that task most activated in the parietal area, and they stimulated that, okay. And what they found was — they did an analysis, and this is basically the alpha, the statistical level. And what you want to do is see when the effect is significant. And this is like how many people it takes to see that significance. You can see that it takes about 47 people if you use a scalp-based method, all right, to see that TMS effect, okay. But it only takes five people if you use the individual targeting method. And this is just an example of being able — that you really want to use this neuronavigation on an individual basis that allows you to most effectively, in terms of spatial parameters, stimulate the brain. And you can increase that precision, by the way. Humans — human operators are trying to hold a coil over the head and keep it right over a head. That's not so easy in itself, even using neuronavigation. And then the subjects receiving the TMS also move.

So, if you're sitting there for a while, even unconsciously, even if you're trying hard not to move, you still move. There are now robot coil holders that stay with the person, stay on target, and position it with millimeter accuracy that allows incredible precision in terms of putting the magnetic field in the brain where you want to put it.

Now, having said that, if you stimulate any part of the brain and you create action potentials in the brain using the TMS, you're actually stimulating whatever that area of the brain stimulated is connected to. You stimulate a network. So even though you have great precision in stimulation, you have less precision in the sense of you're stimulating a number of other areas too. So, you really kind of want to know what you're stimulating in the brain when you stimulate any one part. And there are ways to do that. You can actually use fMRI, for instance. You can use — you can put people in a scanner and stimulate them with TMS in the scanner and using the fMRI imaging, see where in the brain they actually wound up getting stimulated or activating.

This is something that can be used, by the way, to defeat one of the problems of TMS which is TMS itself — here's a coil, okay, at the scalp of the head, all right, and it creates an effective stimulation to a certain depth in the brain. The depth is only right about here into the brain, a few centimeters in that you're effectively stimulating neurons and creating action potentials. But say you wanted to reach a much deeper place, now, for instance, area — Brodmann Area 25, subgenual cortex, is a target now in depression research. But you can't get there directly with TMS, all right? The question is, can I get there indirectly, transynaptically? And the answer we found was yes.

So, if I stimulate a certain place, can I reach this area and aim for it? And you can by doing imaging and doing a thing called diffusion tensor imaging or DTI, and if you see this area of interest, all right, and see what this DTI does is map out the white matter, the wiring of the brain, the white matter tracts, okay. And you can see where they come out to the surface where you can stimulate with TMS. And so, we did that just to show this works. We did that. We seeded it and saw where the white matter tracts came out on each individual person. And then, we put people in the scanner, and using fMRI, we stimulated people in the scanner to see if we could activate that area by activating where those white matter tracts came out. And when we did that — so here's like 10 people. Here are the sites on their heads, in the front of their heads where the white matter tracts came out. Okay, when we did that, we showed that as you increase the strength of the TMS, okay, the dosage of the TMS, you actually increased — oops, didn't mean to do that — you actually increase the activation in Brodmann Area 25. And you can see that here that we actually increased the activation, the number of voxels being stimulated as we increased the strength of the magnetic field, as opposed to other areas of the brain that did not see that kind of increase.

You can also, by the way, look at the — through imaging, in this case, EEG — look at what's happening in the brain over milliseconds, over time. So, if you can use high-density EEG — and I, say, stimulate the motor cortex — I can see its effect in the motor cortex in the first five milliseconds. When I look 15 milliseconds or so afterward, I can see that that activation has moved a little bit to premotor areas in front of that. And in fact, after 24 milliseconds, I can see it's moved — the activation's moved to homologous areas on the other side of the brain. So, I can actually watch the activation happen over time using EEG as well as over space using MRI. And you can see the same thing in the visual cortex in the back of the head. You can see that after 28 milliseconds, it's moved across the corpus callosum and into the other hemisphere, the activation caused by the TMS. So, we are very good at precisely locating where we want to put stimulation into the head. And so, we have a very good control over the spatial parameters.

Another aspect of that is to look at coil shape, okay. Different shapes of coils — one on here, too — for instance, a figure eight coil, which is most commonly used, a double cone coil right here, which is less commonly used but it can get a little bit deeper in the head; a round coil — those are used much less nowadays — metal core coils and the H-coil, which is essentially a very large round coil, produce different effects in the brain. Dr. Zhi Deng will be talking about that in later lectures in terms of the trade-offs that you see, in terms of how deep you can go, versus the focality.

Okay, so, the deeper you go with TMS, usually, the less focal the stimulation is unless you do this transsynaptic trick that I just mentioned before. Another very important aspect is in terms of, for instance, the orientation of the coil — so if you have a figure eight coil, it actually has kind of a horizontal axis to it because it has two loops, okay. And you have a handle here. All right, you can move the handle, all right- without moving the center of the coil- you can move the handle so that the whole shape of the coil moves around. When you do that, that actually changes where in the — what in the brain is getting stimulated and where it's getting stimulated. And that matters a whole lot.

So, the coil orientation is very, very important, too, as well. And that's something that people aren't paying as much attention to but need to. And we're showing that in two different ways here. We're showing that in terms of — don't try to read this table here. But this is 13 people being stimulated with the coil at various angles. And finding how much it takes to stimulate their motor cortex to produce a certain level, what's called a motor threshold, which will also be talked about later in terms of dosing. But, you can — as you change the orientation, you can see that there's an optimum spot, an optimum orientation to stimulate that coil. And it has most to do with exactly how the motor cortex in its gyrus is situated. And what you want to do is stimulate so that the current from the coil is — the current direction is going into the gyrus as opposed to going with the gyrus. And depending on how you turn the coil, that will change the current direction. So, current direction and its accompanying motor — the orientation of the coil matters quite a bit.

Another — I class this parameter in with spatial parameters because intensity determines how deep the stimulation goes and how wide its spread, okay, spatial — which is spatial parameters. And intensity is one of the first and easiest things there are to manipulate with TMS. It's just a knob on the machine, and you can turn it up to 100 percent of the machine's capacity or not. However, intensity — what we found early on, the higher the intensity, the more chance of something like a seizure happening. And so, we — there have been actual studies that have shown, given a frequency that you use, higher frequencies also lead to seizures. And this is the intensity based on the motor threshold I just mentioned before. So, increasing intensity means that you can see that if I stimulate at 10 hertz, you actually have to make the trains of TMS, how long you can stimulate before you risk seizure — those numbers go down with increasing intensity. So, we know a lot about what happens with high intensities. We know a lot less about what happens with low intensities. But, subthreshold intensities actually can also affect neurons as well. And we're just starting to learn about that, too. But that is something we're finding out more about now as well.

So, anyhow, I've gone over spatial targeting. Now, I'd like to talk a little bit more about the temporal parameters that are involved as well, which include waveform, frequency, intertrain interval, and cumulative effects. So, just in terms of the waveform of the TMS — so TMS itself — a single pulse- can look like this particular kind of pulse. The other standard pulse is a bipolar pulse, okay, where the current goes this way, that way, this way, that way again. And so, those are the standard kinds of pulses. Now, I can use these pulses over the motor cortex and create responses in a given muscle, ll right, and I can just get an average of those responses ahead of time. And then, I can stimulate with a different kind of waveform each time. So, for instance, I can use the standard biphasic waveform, which is a basically dampened cosine wave. But I don't have to do a cosine. I can actually use something that's more like a square wave. And it turns out, neurons like square waves better. They respond to them better. And I can also form a square wave, okay, monophasically instead of just biphasically.

So, I can try these different sorts of waves, all right, and then I can test again to see if I created an effect. So, if I stimulate for about 15 minutes with 1000 pulses — say, one a second, okay- I create an effect in the brain that can last on the order of 15 minutes or so before it goes away, it goes back to baseline. This is, by the way — and I'll get to this later — is one of the very standard pre/post protocols that one uses to see TMS effect. So, if I look at the various kinds of waveforms I use across, in this case, 13 people, what happens if I do one hertz stimulation — and I'll talk about this in a second — is that you tend to inhibit. You tend to reduce the response in the motor cortex. And so, I'm expecting a decreased response, all right. This is the people that got the standard waveform in this group that you actually didn't see an inhibition. I didn't see a decrease of response below the baseline. But with the various square wave waveforms, I did. All right, and those will all gradually go back to baseline over time. This is 10 minutes, 20 minutes, 30 minutes.

And so, it gradually goes back to baseline. But the point is that I — that the square wave was much more efficacious and created an effect whereas the standard form didn't. So, the waveform can actually create different effects of TMS. And that's actually something that we're exploring now as well.

One of the first kind of parameters that we found that had very different effects was the frequency. And sort of the rule of thumb that we found was that if you use high frequency, which means greater than three times a second — and we tend to go from — high frequency tends to go from- three times a second up to when you're doing trains, up to about 20 or 25 hertz, okay. That tends to make the brain more excitable. And you can see here with the PET image, that stimulating right here activated a very large amount of cortex when we actually measured the activation of this high-frequency stimulation. Low frequencies tend to down-regulate cortical excitability. They tend to inhibit the response. And so, the stimulation was done about — well, up here — done about here.

And you can see that this blue means that there is a deactivation. And there's — around where we stimulated and here it is — around where we stimulated, there's a deactivation. But that's kind of the rule of thumb that we use. Low frequencies around one hertz tend to inhibit activity. Whereas high frequencies tend to excite. That may have a lot to do with the actual frequencies of the oscillations that are going on in the brain, for instance, the alpha rhythm or the theta rhythm- that are going on in the brain that seem to have a lot to do with function. And this was tested out, for instance, even 20 years — almost 20 years ago — in a study where Klimesch et al. in this case stimulated at the person's own alpha frequency with the TMS over parietal and prefrontal areas while they did a task. All right, and what they found was that the accuracy in the task increased quite dramatically: 15 percent's pretty good. They enhanced the people's response by stimulating at their own alpha frequency. Whereas, when they stimulated at three hertz less than their alpha frequency or stimulated just at 20 hertz, they didn't see this enhancement. So, there may be something to do with how frequent — the reason that the frequencies affect the brain the way they do, interact with the brain the way they do, has to do with the brain's own oscillatory dynamics.

You can look at this in another way, for instance. Remember that — the protocol I showed you with the waveforms, okay, where we do a motor evoked potential, okay. That's the response of a muscle of a target muscle when I stimulate the motor cortex. If I look at the average MEP, motor evoked potential, and get kind of an average of that, and then I stimulate, all right, and then I look at the motor evoked potentials — I'm looking at pre-post changes in that response to stimulation of motor cortex. If I just do the normal one hertz, remember that should just create an inhibition. It did, okay, in this particular study. Here's the baseline. It went — the response after ten minutes of one hertz stimulation did indeed go down and return slowly to baseline. It's back up around 30 minutes or so. If I stimulated at high frequency first — so I do about 10 minutes of that first and then stimulate at one hertz and then measure it again, I created a much more — sorry — much more enhanced inhibition and much more long-lasting inhibition.

So, there's an interaction between different frequencies that I used. Just trying to show here that it's very complex how all the different frequencies actually work with each other. In this particular case, it may have worked because this high frequency would excite the cortex, all right. And then after a little bit, you're –, it's returning to baseline. So the brain is actively pushing back towards baseline, a mechanism that does that. So, it's doing that, and at the same time, I next add the one hertz, which also decreases activity. So, it's already going down, and this shoots it even further down.

So, we're playing with the brain's own homeostatic mechanisms to create even larger effects. Another example of these kinds of interactions is with that theta burst stimulation, TBS, I had mentioned before. Remember, TBS is little bursts of high-frequency stimulation given every 200 milliseconds so that I'm doing it at theta — so that's the theta part of the theta burst. And this patterned stimulation, if I do a continuous amount of that patterned stimulation, for about 90 seconds, I create an inhibition of these motor — if I stimulate motor cortex — I create an inhibition of these motor evoked potentials afterward. If I put a little time in between, if I do a train of theta bursts for two seconds and then I wait eight seconds then do it again and keep doing that for about four minutes, I create an opposite effect, all right. Suddenly, I'm exciting the motor cortex, and it's producing exciting higher amplitude motor evoked potentials. So, I'm able to create opposite effects, inhibition, and excitation, all right, just by adding in a little time between the trains.

So, there's this interaction between the intertrain interval, the duration of the trains, all right, and the frequency of the trains that are very, very complex. And we're just starting to learn how that all works.

Okay. And the last parameter I want to talk about is the duration of the stimulation. So, you can create — and I'll be using this particular slide quite a bit now as I go into protocols — excuse me — is that you can create acute effects. Namely, you can create effects that happen right away in the brain that you can see. For instance, you can — if I stimulate the occipital cortex with a single TMS pulse, all right, and I do it at a certain time after I show say three letters. So, I'm showing you three letters and I'm asking you to repeat the three letters, what they are, and I show those three letters very briefly, like 10 milliseconds.

So, if I do — I lost my — there it is — if I do the TMS pulse, okay — and these are the tracings of three different people. But if I do the pulse at the same time as I show those three letters, nothing happens. People are getting about three letters right, okay, three out of the three letters right. If I do the TMS pulse at 80 or 100 milliseconds after I show the three letters, okay, in other words, the time it took to go from the eye, the retina, back into the brain and back into the occipital cortex is about in this range, okay — then I can actually interfere with people's ability to see it. In fact, they can't report the letters at all. This was originally called a virtual lesion effect. But, the idea is that for — there's a little time period, say for 50 milliseconds, when I can interfere with what's going on in the brain very, very precisely in time. So, I can create what's called an acute effect, a direct effect.

And for instance, I can do that with a muscle. I can make a muscle twitch. That happens right away. I can do this kind of visual masking. I can actually show that I can speed — I can make people better or worse, all right in an immediate kind of way. These effects only last on the order of milliseconds to seconds, okay. There are effects directly on the membranes, the electrophysiology of the brain, immediately. But if you keep stimulating, if you stimulate, for instance, for 10 minutes with say 10 hertz as I showed you before, you can create lasting effects that go on after I turn off the TMS.

And so, if I stimulate for around 10 minutes or 15 minutes or 20 minutes with say 10 hertz, I create enhanced activity say in the motor cortex that lasts on the order of 20 minutes to an hour. And then it goes away. That kind of effect, you're changing synapses, okay, in a way that the synapses are responding to the stimulation and they're kind of adapting to it. And they're actually changing their properties, all right, and adapting to it. And that kind of lasts for a little while until it all returns to baseline. You can make those changes last a lot longer by repeating over and over these 10, 20, 30 minute, you know, bouts of stimulation, all right. And that's what we saw right in the beginning when we were looking at this was weeks of stimulation stimulating every single day for 37 minutes, okay. And you did that from week — after two weeks, after four weeks, after six weeks. And you can see the effect growing and growing. And those effects can last months to years, okay. And what you're doing there is creating epigenetic changes in the brain, and you're creating — in the synapses rather — you're creating changes that last much, much, much longer because you're programming them in through the DNA in a way that lasts, you know, even a year, but basically, at least months.

So, there are a number of temporal scales that TMS can act on depending on what you do and how long you do it that you can look at. So, there are all kinds of levels of cumulative effects that you can create. And this is our great hope for therapy is that we can create these long-lasting effects.

Okay, so that's basically the, you know, the parameters of TMS that we can manipulate, through neuronavigation, through brain imaging, through robotic coil positioning, through realistic head modeling and e-field modeling we can deliver, very, very specific dose right into the brain right where we want it to go. And that allows us to affect very specific networks in the brain, very specific sites and networks that lets us learn, for instance, their functions, give us the ability to diagnose when you can disease populations, for instance, and gives us a really great handle over spatial parameters. Although again, we have a tremendous control over the first millisecond. And then, after that, it's how the brain responds to things. And that matters quite a bit because what the brain is doing at the time and place that you're stimulating, at the point of stimulation, really determines the effect of the TMS.

So, it's not just the parameters that we use externally but what the brain is doing too. There's a big interaction there that we're just trying to learn how to handle and control. And indeed, what happens next is something that we can look at in terms of the temporal parameters. And so I mentioned the TMS waveform and how it really can have affect literally on the membrane time constants, you know, using them and using a square wave, for instance, versus a cosine wave can matter quite a bit in terms of how the neuron responds. Also, the TMS frequencies, how they're related to the endogenous oscillations of the brain can matter in terms of what TMS effects we create. The train duration and the ITI, the intertrain interval rather, can have big effects. You saw that with, for instance, theta burst. Just changing the train — the intertrain interval can change whether things are excitatory or inhibitory. And if you do durations long enough of TMS, you're changing long term potentiation in the synapse. You're changing — you're effecting plasticity mechanisms. Repeating, creating cumulative effects by repeated TMS sessions hopefully create long-term network changes that we can use therapeutically.

Okay. So, given all those parameters, I'd just like to talk a little bit about the basic TMS protocols that we use. And they're kind of divided into two separate kinds of protocols. One is online protocols, looking at acute effects. The other is offline protocols, looking at lasting effects. So, online protocols are stimulating and seeing an effect, okay, right away. These effects are very short-lasting. And it's usually a matter of probing and exploring what the brain is doing at certain points. And it can also be used for biomarkers and diagnostics. And I gave you an example of that already.

For instance, the acute effect here is only a 40 or 50 millisecond effect in the brain- in the visual system using this Amassian study. This is actually, by the way, one of the very first studies ever done- in 1989- in TMS. So, to do these kinds of acute effects, to do these online protocols, stimulating at a certain time, for instance, relative to a task that you're giving people at the same time so you can functionally see changes, you can do single pulses as a probe. You can also do two pulses, all right, for instance, or multiple pulses that are occurring very, very close together in time on the order of a few milliseconds between them. Those produce various effects. And I'll talk about that in a second. Or you can just do short trains of TMS, say two, three, four seconds of a certain frequency of TMS. All these can create acute effects that you can use to study the brain.

The offline protocols, where you're trying to create lasting effects, cumulative effects the best — the prototype example we have of that, obviously is what we see in depression trials whereas we increase the number of sessions, we're increasing the effect — the overall effect of the TMS. And you can see these lasting effects by the kind of protocols you basically use, for instance, in a given session of TMS is pre/post testing. And I've given you a couple of examples of that already where we stimulate — we do, for instance, motor evoked potentials and then, a baseline. And then we do some intervention with TMS. And then, we look at what changes that did to the MEPs. You do a pre/post kind of test. And looking for these kinds of short-term plasticity effects. And you can also do that over multiple sessions and then look at what — look at pre/post, the whole series of sessions. Have you made any changes in people's behavior, in their disease, or for instance, in their imaging and the actual networks of the brain through imaging? Okay, and in fact, the idea in these protocols is well, how do I see a TMS effect?

All right, so I can see it, for instance, online by looking at behavior — so looking in this case in people's accuracy. Or I can do a pre/post effects over time. So, I can actually look at various milliseconds apart, right, and do stimulations at various times and see if there's a difference in effect. So, I can actually do a control in time of this time — oh, there's no effect. This time there is an effect, all right. And just be able to use my statistics that way, for instance. Or I could do these pre-post studies like you saw with looking at the waveform where we do look at MEPs ahead of time and look at their changes depending on what stimulation I give. So, I could look across time in that sense as well. I can also stimulate in different places. So, if I expect to have an effect on say a visual task, all right, by stimulating occipital area, I might stimulate the vertex of the head where there is no visual area, right. And there presumably should be no effect on the visual task that I'm doing. So, I can compare across different sites. That can have a problem because it's kind of hard always to find a place in the brain that isn't somehow involved with your various tasks and behaviors. But those are two different ways to control.

But you can also look across other parameters as well. For instance, you can look across frequencies. And so, you saw that in that PET study where you use high frequency versus low frequency and saw dramatic differences in the brain imaging. Other parameters like intensity, you can do that as well. So, for instance, we did that with the-doing that transsynaptic depth study where we were able to stimulate Brodmann's Area 25 deep in the brain and were able to show a dosage effect that we had control over the activation of that deep area. You can also use — and actually, the first thing people might think of is well I can do TMS and then not do TMS, all right, no TMS and compare what happens when you don't do TMS and when you do. The problem there is TMS itself has its own effects, peripheral effects, all right. When you stimulate on the head, it feels like something, all right. And the more you increase the intensity, the more it feels like something, all right. So, contract muscles on the head. You contract, and you affect peripheral nerves and things like that. It's usually just kind of startling. It feels like a little bit of a thumping. It can be painful sometimes in certain places. But it creates — and for instance, if you hit the right superficial nerve, you can create an eye twitch. You can twitch your jaw. But there are peripheral effects that you create, all right. So, if you do no TMS versus TMS, you're creating effects in one case that can, you know, interfere with tasks that have nothing to do with affecting the brain. So, that's something of a problem. And by the way, the other peripheral effect is it makes a sound, a clicking sound, all right.

So, every time you do TMS, there's always a clicking sound. The clicking sound gets louder with increases of intensity. So, all these things can create other effects, and they're the reason you really don't do no TMS to compare with. What you try to do in this kind of situation is sham TMS. And it has its own problems because how do you create that clicking sound and the other kind of peripheral effects, the thumping, for instance, and all those without actually giving people TMS? And people have been working on that. There are ways you can at least make the sound. You can have a coil on the head and not apply the magnetic field into the brain but still make the sound. And people actually put little electric currents — they put a few electrodes right under the coil and give people a little bit of a surface jolt of electricity right when the pulse happens to kind of mimic the TMS thumping effect, okay. It's not quite the same, but it's similar enough. But there are problems with it, but the idea is to create superficial stimulation that mimics TMS without affecting the brain and then comparing that with the TMS.

So, if you do various combinations of these controls, you're going to actually — you can find ways to compare TMS with no TMS either a — or TMS at a different time or a different place. And in these ways, you're able to pull out TMS effects.

And what are the TMS effects? How do I measure them? So, there are changes in behavior. I mentioned that first- there are changes in performance, for instance, in your accuracy in this case with the visual stimulation or reaction time. That's a good one to use too. There could be changes in clinical ratings. So, the response, as you see here, is in terms of people's ratings in MADRS Scores or in Hamilton Scores in depression. So, you can actually look at those kinds of changes. You can actually look at physical changes that are occurring right away by looking, for instance, at motor evoked potentials. And you can actually do that with EEG and look at TMS evoked potentials right on the head, you know, right underneath, for instance — where you're stimulating you can actually see the response to that stimulation. You can also look at brain imaging changes. I've talked a lot about MRI changes as well as PET changes in this study down here, for instance. You can use high-density EEG. I showed you an example of that as well. So, there's a number of ways you can measure the TMS effects both physiologically and behaviorally.

So, other ways — and I mentioned before other ways to actually stimulate, to do kinds of different protocols that create different kinds of TMS effects, okay. So, for instance, here's an example of an MEP, right? So, I do a TMS stimulation, right, and this is the MEP, the motor evoked potential I might measure in hand muscle that I was targeting. If I do a second pulse, all right, or rather a conditioning pulse, a pulse right before the — that pulse that I did– and by right before, I mean, for instance, like two milliseconds, okay. If I do it two milliseconds before- and this conditioning pulse isn't even strong enough to contract its muscle or create its own MEP, for instance, I can [coughs] — excuse me — I can create an effect. So, this is the response I see, you know, with just a single pulse. And we've traced that again here. If I do a little conditioning pulse ahead of time, okay, I've actually inhibited or quite dramatically decreased the response. If I extend that time, say to 10 milliseconds, I can actually create a facilitation, an increase, all right, against the baseline. All right, so here's a baseline response, and here's the increase that I see, okay.

So, the time, the ISI or the interstimulus interval, that I choose to use can play with what's going on there. What we think we're playing with is the response of different neural systems in a very local location, some that respond to GABA, some that respond to glutamate, so it's inhibitory and excitatory neurons. And you're playing with their dynamics with each other. Okay. And so, you're able to actually look at separate GABA systems and glutamate systems inside one particular area, in this case, motor cortex, by using what we're calling paired-pulse stimulation. And you can increase, for instance, the paired pulses-this is called long interval cortical inhibition-if I stimulate with the conditioning pulse say right here, all right, and with one that's strong enough to actually create a motor evoked potential, and then I wait 50 to 200 milliseconds, I can create — this would be with the test stimulus-if I didn't have the conditioning stimulus I get a nice healthy MEP. But if I have this conditioning pulse, I have a much more inhibited response. That's long-term cortical inhibition. And that's actually — that separates GABA A and GABA B systems, for instance, in the brain.

So, you're actually able to look at different neurotransmitter systems in the same area just by playing with paired pulses and the time between them. And in fact, you can see that, for instance, if I look over a period of one to 12 milliseconds, that I get — relative to the baseline, the test pulse, all right, I get inhibited responses if I have short-term — short interstimulus intervals, ISIs. If I have longer ISIs, I start creating a faciliatory. If I extended this out further, I create inhibitions again and so on. But you're seeing nice and very reliable responses that have to do with neurotransmitter systems locally. And so that's a very useful stimulation. And people do also — you can — you don't have to just do pairs. You can actually do say four, all right, or quadripulse, all right. There's even octopulse stimulation. They're essentially very high frequency, right? If there's only two milliseconds, that 500 hertz, right? But they're very closely spaced stimulations or short trains, all right, and those create their own kinds of effects. If I do short trains of stimulation, this is an example, for instance: if I look at a memory task, all right — so I can show you a letter or — and then I can wait a little bit of time and then show you another letter, and you tell me as fast as you can is that letter one of the letters I just showed you, that's a very typical — it's called a Sternberg task and a working memory task that people use.

Here's another example. I show you six letters, wait a little bit of time, and show you a letter — in this case you'd answer, "No," because it wasn't one of the stimulations. I can do a train, all right, of rTMS, say in that time between the two sets of letters, all right. When I do at the right place, in this case, the parietal area, and I use the right frequency, in this case, five hertz, okay, so the solid areas are active. The lighter areas are sham comparisons, okay. So, we're doing active and sham comparisons here, and we're using different frequencies. So, one hertz and 20 hertz doesn't show very much. This is with a set size of six letters, okay. You can see that I've speeded up in the reaction time two of these letters, all right, if I stimulate the parietal area, okay, at five hertz during this delay period. And that also happens at a set size of one as well. If I stimulate later, like during the probe period, I don't see a difference between active and sham. But I was able to replicate in a different group the same facilitation of reaction time, enhancement of performance, okay, by stimulating during the retention period. So, you can play with trains of TMS, and you can create enhancement for disruptive effects with those trains if they're placed at the right time relative to how the task is going on and in the right place in the brain and with the right frequency.

So, this is an example of an online protocol where you're directly affecting, acutely affecting behavior. And by the way, we're able to show that if we sleep deprive people for two days — and here's a network that activated with the working memory task — and we stimulated inside that network these people that were sleep deprived for two days, and we also stimulated outside the network the same people and stimulated that same parietal area (That's actually inside the network, but you can't see that from here.) -and what we saw was people's reaction time when it was just sham TMS — so again, we're using a sham control, okay, and we're using a space control. We're using different locations, okay. But we see that if I stimulate right outside, nothing happens, right? But if I stimulate inside the network here and here, I actually decrease the reaction time, okay. So, I make people a little bit better and help them recover a little bit from their sleep deprivation. What happens when you're sleep deprived is you get slower, right? And so, we're able to help them recover a bit.

Another kind of offline TMS that you can see — it's a very interesting one — is called paired associate stimulation. In this case, people — again, we're using these motor evoked responses to see this. So, here's a typical one you see by stimulating motor cortex, an averaged one you see. But what I'll do is I'll do 90 pairs of stimulation where I stimulate the median nerve in the hand that's associated with this muscle, okay, and 25 milliseconds later — in other words, the amount of time it takes for this stimulation to get to that part of the brain, okay — and stimulate at that time just when it arrives, okay. I stimulate with the TMS, all right. So, I'm doing paired-associate stimulation. I'm creating a Hebbian-like change of fire together, wire together situation. And I do that 90 times; then I test this area again. I get an enhanced response. Okay, so you can actually change the wiring a little bit. And it's temporary unless again, you do cumulative effects. But you can show these plasticity effects directly. It's a nice way to measure plasticity in the brain, in fact.

Another kind of offline protocol is again that theta burst that I've talked about a few times. So, if I look — and to do a pre/post measurement, okay — that's what you do offline now — is I'm doing a pre/post measurement just like the previous — just like the paired-associate, I do a pre/post with the motor evoked potential Here, I'm doing that again, I'm looking at motor evoked potentials and seeing the changes after I do, say continuous theta bursts or if I do intermediate theta bursts, namely theta bursts where I put that eight seconds in between each two second train. And look at the differences in the post response in the MEPs, all right, with the inhibitory and the excitatory — I mean, sorry — excitatory and inhibitory TBS. So again, looking at theta bursts, you do this pre/post, with the intervention in the middle, measurement. And that's your typical offline kind of protocol.

Another example of that is using, again, a pre-post measure, okay. This time instead of using MEPs or motor evoked potentials, I'm using TEPs, TMS evoked potentials. And I'm looking at evoked potentials of the EEG directly in the brain, all right, when I do this intervention between the pre-post measurements, okay, of stimulation. In this case, it's five hertz stimulation, okay, over I believe it's like 15 minutes, okay. And when I do this 15 minutes of five hertz stimulation, I can see — and this is the pre-TEP responses, okay, averaged in a group and across the brain — you can see that I've enhanced the amplitudes, all right, of the response, okay, after all the stimulation. All right, so again, it's this pre/post measurement that I do when I'm doing this kind of offline sort of protocol.

Here's another example of kind of putting a lot of that together. So, again with that sleep deprivation protocol that I showed you before, this time during the two days of sleep deprivation here, okay, we had four sessions. So, we're trying to create a cumulative effect, right? We're doing four sessions of TMS for an hour each while people are doing the memory task, the working memory task, all right. So, they've done four sessions of it. And then at the end of the sleep deprivation period, we measure — we have them do the task again, all right, with no TMS. So, it's like almost a day later now and no TMS.

So, we're looking for a lasting effect. And so people that got sham TMS — this is reaction time and against baseline, against the pre baseline — the pre-sleep deprivation baseline. You can see that the people that got just the sham TMS, no TMS, slowed down like you usually do with sleep deprivation. But people that got the active TMS look like people that weren't sleep deprived, all right. They get a little bit faster, all right. And they're a little bit faster because they've had a little more practice. But practice didn't help these people with the sham. But it did help people with the active TMS. So, they're not showing the effects of sleep deprivation in this task. In fact, people that are sleep deprived also lapse. They don't answer. They have little microsleeps. And the number of lapses that actually occurred with people with sham was pretty typical. But there were much, much fewer — almost none in most people — in people that got active TMS.

And in fact, the coil was right here over the brain, and the only difference between people that got active and sham was in that area. In that area, that was a more activated area right under the coil that was associated with people's ability to be able to defeat, to be able to — sleep deprivation effect in this task. So, we were able to actually show pretty much complete remediation in this task of the effects of sleep deprivation just by the cumulative effect of using TMS. And so, again, a nice offline strategy, in this case, to try to create a therapeutic effect.

So, overall then, in terms of protocols, all right, you have online protocols that create acute effects. These can be used to both probe and explore the brain and its networks and to diagnose, all right, to look for biomarkers. Repetitive TMS lends itself to both online and offline. You saw the last example I gave was repetitive TMS, five hertz stimulation, and short trains, okay. But you also, with repetitive stimulation and things like TBS patterned stimulation, can create cumulative effects as well. And so, and these are the offline effects where you're looking pre and post something, all right, some kind of change. And you know, I mentioned earlier about the changes themselves. Okay, that actually concludes my exploration of parameters, TMS parameters, and basic ideas, the basic physical effects of TMS, and how we measure them through various TMS protocols.

Original Article

Video » Virtual Workshop: Transforming the Practice of Mental Health Care

Virtual Workshop: Transforming the Practice of Mental Health Care


Watch on YouTube.


Mi Hillefors:
For your information, I've started a recording now, Holly.

Sarah Lisanby:
Justin Baker, I just want to check your audio.

Justin Baker:
Any luck? No audio?

Sarah Lisanby:
Yes. And just checking if Joshua Gordon's audio was working? So, good morning everyone, and welcome to the NIH sponsored workshop, a virtual workshop on transforming the practice of mental health care. And I want to give a few minutes while our attendees and panelists sign on. My name is Holly Lisanby, and I really am delighted that all of you are able to join today. I see now that we do have Joshua Gordon signed on. And Josh, just wanted to test your audio.

Joshua Gordon:
Hi. Good morning, everyone.

Sarah Lisanby:
Good morning. So, I've just opened the workshop, and I want to welcome everyone to this workshop. And to get us started, I'd like to introduce the director of the National Institute of Mental Health, Dr. Joshua Gordon, who is going to give us some introductory remarks, and then I'll come back with some format and goals for the workshop. So, welcome, Dr. Gordon.

Joshua Gordon:
Good morning, everyone. Welcome. Thank you for joining us today, both the panelists and the attendees. I'm really excited about this workshop. It certainly will hopefully be a break from the humdrum every day. I know that most, if not all, of you are, like I am, working from home, and that your lives have been quite disrupted by what's going on. So, I want to acknowledge that first, and to thank you from — for taking time out of your schedules, and for dealing with childcare, whatever else you have to deal with. And we understand if there's a bark in the background, if there's a kid that comes, you know, and needs something. So, don't worry about that. But we're really pleased that you could all join us today.

One question you might have is why are we doing this? Why is the NIH having a workshop on transforming the practice of mental health care? This workshop is meant to be really, truly forward thinking. We're trying to imagine what the practice of psychiatry could be like, if we took advantage of the tools that are coming online now. We know that we could do some pretty amazing things, in terms of the understanding and treatment of mental illnesses, if we can harness these tools. What kinds of tools ma I talking about? I'm talking about the ability to get broad data sets from a number of different features about our patients, be they be features of the brain activity, features of their past experiences, features of their longitudinal course.

If we can harness all these data — oh, and I suppose my geneticist colleagues would kill me if I didn't say features of their genetics, as well. If we can put all this data together, we can learn a lot more about our patients, learn about a lot more about the illnesses that they are — that they are struggling with. And, potentially, move from a practice of a psychiatry, which is based on self-reported symptoms and based upon best guesses on the part of the clinician and the patient as to which treatment options are going to work best for them, are going to be most tolerable for them, et cetera, into a data driven practice, where we think about patients, yes, from the personal narrative perspective that will always be part of psychiatry, but also from the perspective of what's going to work best for them, and what's going to give them the greatest chance of feeling better and the least chance of side effects.

My model in thinking this is the modern day oncologist who gets as much data about the cancer that her patient is suffering from, and gives that patient a series of treatment options, each of which is accompanied by a probability of success, and also, a probability of side effects. And, then, with that information in hand, the patient and the doctor can work together to decide which treatment is most appropriate for that particular patient in that particular situation. This kind of model of the practice of psychiatry will require, number one, a lot more research. We need to be able to, obviously, answer lots of questions about our patients, about — both in aggregate and in individual in order to be able to imagine this kind of psychiatry.

It will also require an infrastructure. It will require large data sets that are integrated and that have, both, depth, in terms of depth of data about individual patients and breadth, in terms of large numbers of patients. It will require knowing what kind of data to — that goes into these data sets. And, finally, it will require a decision engine. It will require an infrastructure that will allow psychiatrists and patients to put data in and an algorithmic infrastructure that will process that data and give probabilities and likelihoods back to the psychiatrist, in order to aid the patient. So, that's the big picture we're talking about.

We're imagining a different kind of mental health care, one that is patient-centered but data-driven. And we're trying to think what do we at NIH need to do? How can we take what is really a wonderful revolution of new technologies and really bright people, many of whom are assembled here today, who are working with these technologies, to learn about our patients? But how — what can we add to this mix to bring this future potential into reality? And, finally, I'll just say what I said at the beginning, which is this is a very forward-thinking workshop. We're not imaging that we can accomplish this mission in the next six months, in the next five years, maybe not even in the next 10 or 15 years. We're really thinking forward, but we're recognizing we need to lay the groundwork now to be able to achieve that vision in the future. So, again, I really want to thank you. I'm very much looking forward to the day, and I'm going to turn it back over to Holly.

Sarah Lisanby:
Thank you, Dr. Gordon. And I'd like to thank you, Dr. Gordon, and Dr. Shelli Avenevoli the Deputy Director of NIH. And, also, I'd like to thank our moderators, who have graciously volunteered to help organize the content and the discussions we'll be having today. The moderators for our first session on the future vision of where we might be able to bring mental health care through the advantages of data and technology, specifically, Dr.'s Andrew Krystal and Justin Baker. I'd like to thank our moderators for our second session, which will take us through learnings from existence networks and efforts underway, [inaudible] Doctors Guillermo Sapiro and Helen Egger. Our third session moderators will help us understand the challenges of getting from where we are today to where we want to go in the future. That session will be led by Dr. Jenn Wilsack and Jeff Gerard. And our fourth session, on ethics and special populations, co-moderated by Doctors Deliete Justee and Jeremy Veenstra-Vanderweele. And, I'd also like to thank our program committee, starting with Monica Carter, who is a known to most of you because she really made this happen, and really is doing the lion's share of the work here.

I'd also like to thank the acting Deputy Director of the Translational Research, Dr. Mi Hillefors, and the members of the planning committee here at NIH, Dr.'s Jenny Pacheco Susan Azrin, David Leitman, and Adam Thomas. I'd also like to thank our speakers. We know that devoting most of the day is a tremendous ask of your time. We've invited you because of your complementary expertise that will help us think through where research can make the most impact in transforming mental health care. And, as Dr. Gordon acknowledged, we understand that the world has changed since we first organized this workshop. And we understand, as a consequence, some of you will have to be leaving early or not participating in certain parts of this. And we want this to be flexible. Please — we understand that you need to come and go based on your time schedules and challenges.

So, the purpose of our meeting, as you heard Dr. Gordon explain, is to assemble a group of experts to help understand how multiple streams of data, including electronic health records, biomarkers, genomics, digital tracking, physiology, and imaging, can be used to inform treatment selection in mental health care in the future. A key, deliverable from this could be technologies such as clinical decision support, software tools or other aids, to aid treatment selection. We've designed this program today to hear from a large number of experts from diverse disciplines in a short period of time, to help us shape our thinking on these topics at a very high level. The format of our workshop; it is open to the public and you — this will be recorded and archived in a publicly accessible format. For those of you who need it, we do have live captioning, which is available in the multimedia viewer, in the lower part of your screen. You do need to click on that to activate the live captioning.

A few housekeeping notes: as you join, please mute your microphones. We understand — we do have a brief break in the middle of the program. But we don't have a formal lunch break or meal break, so please do take breaks when you need to. We're also having attendees from various time zones, and so that resulted in some aspects of our time frame here. Please note that there's a chat function and a Q and A function on WebEx, which you can use to direct to the panelists during the discussion sessions. So, those are actually all of my introductory remarks. And, so, now I'd like to turn to introducing our first session.

So, our first session is focused on the desired future state, what we — what problems we hope that new technologies and data-based approaches can solve for us, to improve mental health care from the perspective of the patients, providers, health systems, and payers. And, so, I'd like to introduce the co-moderators for Session Number One. Dr. Andrew Krystal is a professor of psychiatry and vice-chair for research at the University of California, San Francisco. He's an internationally renowned expert on sleep and mood disorders, and he's joined by Dr. Justin Baker, who is Scientific Director of the McLean Institute for Technology and Psychiatry, and the Director of the Laboratory for Functional Neuroimaging and Bioinformantics at McLean Hospital. Now, I'd like to turn it over to Dr. Krystal to kick off our first session.

Andrew Krystal:
Thank you very much, Holly. I just want to make sure I can be heard. Thank you. As Holly mentioned, the first session is about the big picture, what it is that we hope will be achieved in the future, and what this will do for treatment of people with mental health problems. And we're going to be looking at this from the perspective of some key stakeholders in this. That is, the providers, health system's payers, and you — we have brought together a group of distinguished scientists to help us with this process. And each one of these individuals will be presenting one slide and will have two minutes to discuss their perspective, and myself and Justin Baker will be moderating and keeping everybody on track. And after the presentations from our experts, or the panelists, for session one, we'll have a time for question and answer. Question and answer will — questions will be entered using the chat function of the web interface. Now, I want to turn it over to Justin Baker, who will start us introducing the panelists for session one.

Justin Baker:
All right. Well, thank you, Andy, and thanks everyone, Holly and Josh who are hosting this exciting event, which I'm very pleased to be part of this morning. I think we all recognize that, in this time, especially, we're faced with new access issues for mental health, and though the visionaries you're going to hear from today are really charting the course for the — that desired future state for psychiatric practice. And, so, we're going to hear from each separate speaker here, this morning. Very briefly, each speaker is going to present a single slide representing their vision for the future of psychiatric practice. And we're going to start here with Terri Gleason, who is the Director of Clinical Science and Research — Clinical Science Research and Development Service, in the Office of Research and Development at the VA central office, in Washington, D.C. Dr. Gleason has a distinguished research career of nearly 20 of VA service and has held many leadership positions across the government. And today, she is going to be speaking to us about her vision and the VA's vision for where psychiatric practice is headed. So, Terri, please.

Terri Gleason:
Thank you, and good morning everyone. I'm just delighted to be here. And, yes, it's an extremely busy time for us in VA, but I think it sets us to be just right in terms of thinking about visioning and, actually, a way forward. As Dr. Gordon described, a way into the future and how we get there. So, the place that I sit in VA is the research office, which is embedded in the national health care system. And what that means, of course, is that we're in a unique position to generate questions that need to be answered for our veteran population, drive the research for them, and bring them back the results right into the health care system. So, the depth and breadth of data and the richness of data that can be generated, if we are greatly organized and forward-thinking, means that the potential is just, possibly, boundless. So, there's so many resources that can be brought to bear, in terms of thinking or conceptualizing a learning health care system.

So, right now, today, you know, we're in the midst of this crisis. I saw numbers yesterday that reflected 1,600 cases in VA of confirmed novel coronavirus cases. And, unfortunately, we're at a level of 53 deaths, as of yesterday. These are, you know, our veterans who have served, and I think it's a really important point about how sobering the moment is. On the other hand, I do want to bring to the group that what that means is we can actually capitalize, now, on the learning lessons that we're going through as a system, and for research, that just makes this conversation all so timely. So, what I'm hoping for — it's not reflected on my slides, but what I'm hoping for is, in this vision and moving forward, we can take all the learning opportunities of things that seemed like challenges, whether it was related to regulation or policy, and so forth — the delays in research, the using of data, and so forth — we can capitalize on that and turn it into a situation where we can make progress faster, because now we know it can actually happen in real-time. So, I did just want to make that global note. Our AI team, are busy, working 24/7, data scientists across VA, to capitalize on all the data that we need to draw conclusions for the coronavirus, and, in the same regard, we can eventually put that experience to work for mental health in other scenarios, as well.

So, just briefly, I'll walk through, I mean, the vision for mental health research in VA — is it — as I said, a diverse system, and I'd like to take into account, at the patient level, we need the best evidence base, and that's obviously produced by rigorous science and using everything that we can. I think it was already listed, but all of the richness of data that can be brought to bear for the patient, and directing that, and what that means, you know, from the time they walk into the door, until the treatment plan is developed, and then the response is actually known and recorded in a real way, and accessible to research. For the providers, we need evidence-based treatments, and we need the level of knowledge that they can use that takes in consideration patient and system level factors, to derive the best treatment plan for our patients. I want to bring in comparative effectiveness research questions, because I think that should be a part of the vision and how that could work in the future, as a tangible, I think, approach that we can actually make progress on quicker if we all get organized around that. And, then, of course, at the systems level, talking about what a learning health care system really needs and, as we capture data at the entry point for patients and move that through research level analysis, and bring it back to health care, implementation is really the complete picture, I think.

The challenges, very quickly — all of — all of the above, I mentioned, including data integration that starts from the beginning with a common set of assessment tools — perhaps not so much leaning on self-report, but considering what we can bring in terms of organization and standardization of data, across the system to begin with. So, I'd like to stop there, just keeping note of my time, and I'm happy to stay at least through this panel with you today. Thank you, again.

Justin Baker:
Thank you so much, Terri. Yeah, speaking of time, I'm going to — I'm going to, actually restrict — let's try to restrict introductory remarks to just a couple of minutes for the — each of the initial speakers. I mean, what we'd like to do is broaden it up to a discussion among the speakers for the remainder of the hour. So, if that's okay, I'll go ahead next to David Ledbetter.

David Ledbetter:
Yes. Can you hear me?

Justin Baker:

David Ledbetter:
Okay. Thank you. I'm glad Josh got in a quick mention of genetics. I'll try to summarize where we are in genetics with a future vision that will have complete genomics information on every patient in a clinical setting and every patient participant setting. Our goal at Geisinger is to have complete genomic sequence information on every baby born, as part of routine newborn screening, to inform health care throughout the lifespan and to allow us to prevent disease, rather than always reacting to disease. So, at a high level, for many common adult diseases, there's evidence from twin and family studies, a very high heritability, or a significant genetic contribution. These range for things like cardiovascular disease risk, obesity, diabetes, many other common diseases, from about 20 or 80 percent, or heritability estimates pf 20 to 80 percent, and, on average, 50 percent. So, the pie diagram there — and you can adjust the relative sizes of each category to your preference. But I'll use 50 percent genetic risk factor. What I like about genetics is the human genome is big. But it's finite. We now have a very good map, and we have tools to measure, accurately, the DNA sequence of the entire genome of all our individual patients, all individual research subjects. And I think that allows us to, sort of, remove one degree of freedom and complex, multi-factorial traits, on a clinical and a research basis.

To the right, I've shown the two best characterized psychiatric disorders, in terms of common genetic variants, which comprises the heritability index. Coming from GWAS studies, but now being translated into polygenic risk algorithms to predict risk of disease based on hundreds and thousands of different small effect contributions. In autism, heritability is among the highest of any disease known, at .9, or 90 percent; for schizophrenia, also quite high, at .8. And people are working on developing polygenic risk scores, in order to predict risk of disease for these disorders. For rare variants, of rare mendelian genetic disorders, a copy of number variants or single nucleotide variants, in pediatric autism, depending on the cohort, up to 25 percent have a single large effect size rare variant that's the main contributor to autism.

In schizophrenia, a much lower yield of about 5 percent, with 22q11 deletion syndrome being the most common and well-known, well-described risk for schizophrenia. Below that, I've shown that to get this genome wide SNP data is now about $50. And the algorithms for developing polygenic risk analysis are getting better and better. People in the field are not using them and planning to use them as stand-alone but incorporating them into other disease prediction risk, the most advanced in cardiovascular risk prediction and some cancer risk prediction. Whereas, DNA sequencing to identify rare variants is still in the $500 to $1,000 range, even at scale, for exome or genome sequencing.

The bottom conclusion is stratifying cohorts by genetics decreases heterogeneity; I think will increase power in identifying important environmental life style factors contributing to risk, and help us to more efficiently identify effective interventions and treatments, individualized to patients genetic background or the presence of a rare genetic disease. So, the goal, actually, is for clinicians and researchers that you won't have to think about genetics. We will have genetic data on everybody — every individual patient, every research subject. And that will be built into predictive algorithms, other models, characterizing cohorts of patients for clinical use and for research. I'll stop there.

Justin Baker:
Excellent. Thank you so much, David. That's exciting. So, next, we're going to hear from Amit Etkin and Corey Keller. Amit is the founder and CEO of Alto Neuroscience, and he's a professor currently on leave to Alto, in the Department of Psychology and Behavioral Science at Stanford University. Corey is a professor in psychiatry and behavioral sciences at Stanford, and also the Chief Medical Officer with Alto. So, Amit.

Amit Etkin:
Great. Thank you for organizing this event. Let me just say, very briefly, you know, one view of the future in psychiatry, which is really to try to break as much from the past as we can, seeing what — oncology was mentioned earlier — is out there, as an example. So, you may have seen, when the innovation was coming by there, some articles there on the left — 2008, 2018, the results from very prominent meta-analyses, where in 2008 Irving Kirsch looked at all of the antidepressants that were submitted to FDA and did a meta-analysis, not just published but unpublished data, and found that there was a small effect size between drugs and placebo, and the conclusion in the laypress [spelled phonetically] was that these drugs don't work.

Andrea Cipriani, in 2018, did pretty much the same thing with more advanced methods, and came out with a conclusion, that the same newspaper memorialized, is, in fact, the drugs of working. But, in reality, both of these actually had the same exact effect size, and this was meant to illustrate our own change, going from, in 2008, feeling like these drugs work and then our hopes being dashed, and now our hopes being so low that any evidence of effect is reason to celebrate.

My argument is really that perhaps this is a misplaced focus. And our focus should be on understanding why the effect size is so small in the first place. And perhaps the future state — our desired future state is just completely end one-size-fits-all psychiatry, in terms of treatment development and use, and really adopt what oncology has done, which is really a more precision approach for everything.

So, let me give you an illustration of how this could work. This is from a recent set of papers we've published on NIH sponsored study called EMBARC. EMBARC is an antidepressant treatment prediction study that took 300 depressed patients — you can see, in the panel there, the left-hand graph — gave them sertraline and a placebo, and sertraline was found to be nominally better; 42 versus 34 percent response rate. But not very impressive. What we did, though, was ask the question of is it the — that the drug is not effective at all, or the fact that our diagnosis is imprecise that’s really the issue? So, we developed a machine learning — a novel machine learning method, trained on EEG data, given prior to — acquire, rather, prior to the treatment, and found a pattern that was strongly predictive of sertraline — could differentiate that from people who respond to placebo.

And, if you look at the figure on the righthand side, this is snow stratifying people based on their cross validated EEG based prediction of sertraline response. And what you can enter percentiles, the lowest quartile, middle quartiles, and upper quartiles — and what you can see is that, very clearly, only a minority of people are responding to the drug. They have the highest quartile — based on EEG, has an almost 70 percent response rate, double that of placebo. So, there's important lessons there, not only that there are people who respond really well to these drugs, and these drugs are actually very effective — these are — imprecision in diagnosis, that's the issue — but that we're driving the effects of drugs or device based intervention studies based on tiny subgroups out of the whole population of patients. And, so, the hope for the future is really an end to current pharma as it is in psychiatry and opening the door to how pharma is in oncology, which is developing a drug for a group of people based on an understood approach and set of patterns. I'm going to stop there let Corey take over.

Corey Keller:
Thanks, Amit. I'm just going to say a few thoughts about the challenges for implementing biomarker research in clinical care. When we think about our future state, we really aspire to have our research be real world, multi-modal, large scale, and longitudinal, with minimal patient burden. This will allow us to develop novel. powerful biomarkers and treatments. Unfortunately, it's not that simple, and implementing biomarker research is really a balancing act. When we think about real world research, we want our research to be as generizable as possible, but in the real-world high complexity of disease and comorbidity. The keeping it scientifically rigorous is not a simple task. Do we utilize strict versus loose inclusion exclusion criteria? Do we undergo naturalistic data collection or randomize clinical trials?

When we think about multi-modal, which biomarkers do we use? There are many at our disposal and many that relate to mental health, including EEG, MRI, genetics, passive digital biomarkers. The list really goes on. When we think about patient burden, how do we maximize the frequency of the assessments and quality of biomarker collection, while minimizing patient burden? And, finally, analytics. How we do combine different biomarkers with different time scales, sizes, and dimensions together, in a meaningful way? In all, for the future state, finding that right balancing act is critical to keep scientific rigor and patient satisfaction high, while minimizing costs and patient burden.

Justin Baker:
Excellent. Well, thank you so much, both of you. It's very exciting. Next, we're going to move on to Timothy Mariano. Timothy is a medical director in late stage clinical development, as Sage Therapeutics. He's also the associate investigator and staff psychiatrist at the Center for Neurorestoration and Neurotechnology at the Providence VA Medical Center, in Rhode Island. Tim.

Timothy Mariano:
Thank you. It's a — it's a pleasure to be here with all of you this morning. So, I'm going to talk about two interrelated aspects that I think will be predominant in the future, involving digital mental health tools and, also, rapid treatments in psychiatry. And, really, that underpins your two aspects of what I think our future vision, our future state, may entail. I expect that we'll see a movement away from more traditional, and, if you will, low temporal resolution measures, such as rating scales that can capture a patients state at discrete timepoints, that may be weeks or even months apart, and that those will be replaced with digital biomarkers, including continuous biobehavioral data. And this will be backed by a near ubiquity of the hardware, including not only mobile devices, such as telephones or fitness trackers, but also integration into everyday objects, which might include jewelry, clothing, even medication tablets or packaging for food products, all providing the potential for data that must be weighed, of course, with appropriate ethical considerations.

In line with this, in the future, we'll probably also see a movement away from traditional antidepressants that may take on the order of two months to show effect to more rapid acting treatments that can demonstrate meaningful clinical difference in, perhaps, days to weeks. And these could include noninvasive brain stimulation technologies. I give a couple of examples on the slide of current and emerging ones. But also, novel pharmacologic agents, such as esketamine and brexanolone, too — recently approved rapid acting antidepressant treatments.

The eventual AI ecosystem will be challenged to juggle stakeholder priorities that may, at times, be competing. I tried to break this down along a couple of dimensions, here. Treatment developers — that will be the scientists and companies that are involved in developing these treatments — will need rapid predictors of response or non-response to their candidate modalities. Today, we do use digital health tools and clinical development in the biopharma space, but oftentimes, they're included at phase three or post marketing, which I would argue is too late in the process. It really should be integrated much earlier. Also, today, inclusion of those tools usually adds substantial burden for sponsors, for sites, and for patients and participants. There's usually an additional app that must be installed, additional devices that must be carried, additional visits, additional data entry. And, also, competing goals for what the treatment developer wishes their eventual device or technology or medication to have in the marketplace.

One is, you know, with additional health tool, is it a purely diagnostic, a therapeutic tool or some combination of both? And that must be weighed against the candidate treatment, and whether that will purely a monotherapy treatment or an induction or an augmentation treatment. All of these would affect how we might want to integrate such tools. Providers and payers, of course, will want an early warning system provided by the AI ecosystem to identify patients who aren't doing well and to suggest alternative treatments. And, last but not least, patients themselves will want an AI ecosystem that not only provides some information that they might be doing worse. That might be important in diseases such as schizophrenia or bipolar disorder, where sometimes patients do not have complete insight when they are starting to do less well. But patients may also want some therapeutic option; for example, some provision of stopgap treatment to bridge them until they're able to see their provider, either virtually or in person. And that can comprise web-based CBT bots or online group teletherapy, to give two current examples. Thank you.

Justin Baker:
All right. Thank you so much. And, now I'm going to hand it over to Andy to introduce the remainder of the speakers, and then we'll move on to a discussion of the future vision. Andy.

Andrew Krystal:
Thank you, Justin. Our next speaker is Brian Shiner. Dr. Shiner is a staff psychiatrist at the White River Junction VA Medical Center and an assistant professor at Dartmouth Medical School, in the Dartmouth Institute for Health Policy and Clinical Practice. He's been involved with research at the VA Medical Center, evaluating the effect of new primary mental health care clinic on access and quality of depression care, and has completed multi-site research using performance measurement and interview data, to evaluate clinical processes that promote higher quality and improved efficiency. Dr. Shiner?

Brian Shiner:
Yeah. Dr. Krystal? Hello?

Andrew Krystal:

Brian Shiner:
All right. You can hear me now? Okay. Great. So, thank you, Dr. Krystal, for using a bio from 10 years ago because that gives me a picture from 10 years ago when I had hair. So, that's really pleasing to me. My desired future state for the health care system, and for the VA in particular, where I work, is that we leverage the data produced in routine practice to learn and to plot our path forward. And I'm not the first person to say that. The IOM and the National Academy of Engineering, 15 years ago, said that in their report and called it a learning health care system. And, certainly, the IOM CER report a few years after that kind of said the same thing, about how we should use health care data.

I think that, for me, in the work I do as a psychiatrist, I think there are two things that we can do that are important. One is to kind of look at current practice and see how it works in the real world. And currently, I'm wrapping up a DOD grant, where we're comparing five medicines that we know were effective for PTSD in clinical practice, using patient reported outcome data, as well as pharmacy data, and data from psychotherapy notes on what treatments they're getting for PTSD at the same time, and, looking at not only the effect overall, but in specific clinically important subgroups, such as women, people with traumatic brain injuries, and so forth.

So, from that work, we're kind of moving into a new area of research, which I hadn't really imagined up until about two years ago, when I started to write a R01, with my colleague Jamie Gradus. We're going to use the data that we've developed for our DOD funded study, which is about two million patients with PTSD in the VA 2000 to 2019, about 20 years of data, 500,000 have pre-post outcomes information about their — about their PTSD called the PCL. We're going to look at about 2,000 drugs that are FDA approved. We'll be looking using a method called tree scan, developed by our colleagues, Martin Kulldorff and Krista Huybrchts, down at Harvard, to scan for drugs and classes of drugs that may actually be effective for PTSD that we don't know about.

The unique thing about our tree scan is we'll be considering drugs in terms of mechanistic classes, you know, such as alpha to antagonist or whatever, rather than clinical classes, such as, you know, anticonvulsants or antihypertensives. So, we'll be looking at mechanism rather than what they're used for. My colleague Jang Wi [spelled phonetically] has been working with me for many years now on the DOD work, where we're comparing the currently approved drugs. As we discover things with tree scan, we'll be comparing the drugs we discover with tree scan to drugs that don't appear to be effective for PTSD but are used for the same thing, in order to get somewhat, at our causal analysis. And, then, of course, once we're done with that and we find our top candidates, we'll be doing the same thing to look at safety of the drugs that we think should be studied or used moving forward.

So, that's just an example. I know there are lots of different examples and thoughts about how to learn from health care data, but this is kind of how we're doing it. So, just to give you a sense, I think there are — I wrote two challenges. I think there are actually three. The third one I can't write down. But, so, one big challenge is that the VA is a unique computing environment. So, we have data on patient reported outcomes that have been collected since about 2008. We have really very granular information about prescriptions and when they were picked up, when they were mailed, what the doc — what the doctor said in the comments bar, et cetera. And we also have access to all notes written in psychotherapy encounters, so we can train computers to read those notes and say whether people got, you know, psychotherapy that's effective for PTSD while they were getting these medicines, for example. So, we can account for confounding a little bit better.

So, you know, whether that kind of data exists outside the VA is a big question on my mind. I haven't heard of other health systems having all that, although I'm sure they will soon. It's also been very challenging to share our data sets with people outside of the VA. The VA — it just has a lot of challenges around making sure we're protecting health care data at the same time that we're learning from it. So, I think that's — our colleagues at VINCI, which is the VA data repository. I've been working on that for several years, and hopefully, there will be something coming soon.

The last thing is the VA's about to go through, or is beginning, right now, a transition from our homegrown EMR VistA to a contract with Cerner, and, you know, things may be better in the end, but for — you know, for a period of as many as 10 years, there may be holes in the data, and the data that we end up with in the end might be better, but it might also be different. So, we might able to ask different kinds of questions, but perhaps not the same questions we've already been asking. Thank you.

Andrew Krystal:
Thank you, Dr. Shiner. I just want to remind our speakers, because we're running a bit behind on time, to try to keep their remarks to two minutes, maximum. I know there's a lot of exciting points to raise here, and we'll all have – hopefully, have plenty of time to bring them up in discussion. Our next speaker is Diana Clarke. Dr. Clarke is an experiences epidemiologist research statistician, an educator with a long history of working with academic institutions, specialty organizations and behavioral health agencies and hospitals. Please, go ahead, Dr. Clarke.

Female Speaker:
Dr. Clarke, you may be muted.

Diana Clarke:
Sorry about that. Yes, I was. So, I'm going to jump right in, then. One of APA's mission is to help psychiatrists and other behavioral health providers provide high quality care to patients with mental and substance use disorder. And we know that their variability in where these providers are, where they practice — variability in terms of how they practice, and then, also, to patient population they serve. So, they have variability in terms of access to different types of resources.

And, so, it was part of that that actually led to us, you know, actually developing this national mental health registry that we've developed that APA calls PsychPRO. And the goal for PsychPRO was to kind of bring together data from information across areas – practices, whether it’s solo practices, academia, emergency, outpatient, inpatient, and primary care. Because we know that majority of found patients with mental and substance abuse disorder do present to their primary and get majority of their care from primary care. And, so, it's trying to bring together all of this data, whether it's data from electronic health systems, data from patient portals. And, also, it was this some grand idea that we'd also try, for patients that agree to do this, actually also bringing data from any kind of wearable technology.

And the promise of this is, if we can actually bring — get all of these types of practices and different settings — to actually join PsychPRO and bring in all of this data, the promise of that would be that with using all of the big data analytics, we would be able to share data, not only with the patients who are providing this information — because a lot of — in terms of speaking with patients, we find out that sometimes they have a problem with providing a lot of data and getting no information back. So, a way that we can actually share information with patients, but then also use that information to actually develop practice guidelines, to actually create or identify effective measures of psychopathology, so we can actually have improvement in our diagnostic manual, or any kind of diagnosis for mental health and substance use disorder.

And, then, also use that information to provide real-time decision support and real-time interventions to the clinicians, regardless of where they are and where they're practicing. And, the hope is that this type of integration of different health systems and different data types and patients from various part of the country, we could actually enhance the quality of care to mental health and substance abuse patients. And that would be because the idea is to create the system that — system learns, provide information to patients and to providers. And as we get information from them, in terms of how things are working, we identify where quality of care is great versus poor, and then create learning opportunities for providers, and, actually, also provide resources and information for patients as well.

So, you know, some of the challenges that we do see with this is — and I've listed some here, but one of the biggest ones is actually challenges in bringing on some of the largest settings, especially academia, where you get into the issues — legal issues, et cetera. And it's not just the legal issues, because we've worked out most of those. It's where — the trust and sharing data. And I know the person before talked about sharing data outside of the VA. And, so, it's kind of overcoming some of those barriers, and then also overcoming barriers with the different electronic health systems, and whether they're willing to allow us to integrate with their system, to pull data into their register — but not just to pull the data in, because the usefulness of this would be to actually push the data back — information back to the EHR, so commission can use that information in their clinical care.

So, that's how — one of the — a couple of the challenges that we've actually faced. We also know that the [inaudible] challenges, as well. I mean, as part of this registry — because it was built to provide — as a quality improvement tool, and to be used secondary for research purposes. Then we get into thinking about the ethics of “Do we need patient consent?” And because it's primarily for quality improvement initiative, we don't necessarily require patient consent to actually use the data to inform those things, where we do require — based on the common rule — where we do require patient consent is to get patient to consent to be re-contacted for future research studies.

And — but the — it has great promise, in the sense that it creates a repository of data that can be used to answer a lot of interesting questions; can, as I said, drive revision to diagnosis and form practice guidelines. But it also provides a repository of patients who have actually completed consent to be re-contacted and can be available to participate in different types of research studies. And we know one of the limitations is that many patients with mental and substance use disorder doesn't necessarily make into a lot of these research studies that have been done to date. And, so, I'm actually going to stop there, just because I'm trying to make sure I stick with the two minutes, and I'm hoping I didn't rush too much there.

Andrew Krystal:
Thank you very much. Our next – sorry. Thank you.

Female Speaker:
Those who are not speaking, please mute yourselves.

Andrew Krystal:
Our next speaker is Dr. Neal Ryan. Dr. Ryan is an expert in pediatric mood and anxiety disorders. He's led large scale studies of behavioral change in children — adolescents, diagnosed with mood disorders and has investigated the effectiveness of therapies; currently, the principal investigator in a large NIH funded project related to the neural biology of childhood anxiety and depression. Dr. Ryan.

Neal Ryan:
Hi. Thank you. Let's try for the two minutes. Look, what I want to say today is that social determinants of health really matter, and you can get those only by natural language processing of the chart. So, if you've got high value predictions, you're going to have the stuff that we're talking about and talk more about, but you probably want to add the social determinants, as well. You know, losing your job, becoming homeless, having marital conflict, imprisonment, and so forth, still matters. I'm not seeing my whole slide here, by the way. Anyway, so all of these are hidden in pretext notes in the medical record. They're not in structured data, by and large. You know, they're in social work notes, they're in emergency room notes, in the pretext.

How's this been done in past? Well, there's several things that are too simple to work well, like bag of words or worded bags. There's a — structured ways that work well in medicine, like cTAKES; cTAKES — excuse me — clinical text analysis knowledge approach. And then there's the new approaches that started two years ago, with the deep learning language models, BERT and ULMFit and so forth. And those are far and away the best the way to understand sentences, right now, that's available, in any sort of machine learning. And I think they'll become much more important in the things that we're going to be talking about today.

The example that I wanted to give is David Brent, Rich Tsui, and I looked at predicting suicide attempt after an ER hospital visit — a high value thing. You do as best prediction you can. Using all the structured data and looking, say, for example, at the seven day — suicide attempt in the next seven days are really important when you've got somebody in the emergency room — you know, do you hospitalize them? Do you do — give them a really near-term appointment? What do you do? We've got an AUC of 0.88. As you guys, mostly, will know, and AUC of 0.10 is, you know, and oracle, the gods predicting. An AUC of 0.5 is completely random prediction. So, 0.88 is pretty nice prediction. When we added the natural language processing data, it went up to 0.92. So, that's a quarter of the distance to perfection. You know, that's actually — I'm sorry, that's 33 percent of — so, going from 1.0 to 0.88 — improving that by 0.04 is improvement of a third of the way to perfect prediction, from what was already good prediction. The things that mattered in that one were marital conflict, imprisonment, family support, going into rehab. So, lots of good and bad things that were not in the chart, were not in the structured parts of the chart, were in the pretext.

Why is this hard? What are the challenges? Well, there's some technical ones, like completely anonymizing things, getting access, idiosyncratic abbreviations, local abbreviations. But the really hard one is, is that the best techniques, the deep learning language models, are exactly two years old — in the world, not just in psychiatry or medicine. And, so, we don't completely know how to use those as well, and we don't have them trained up. I think that, for other high value predictions, like readmissions, like starting to use — starting to abuse opioids after you give them an opioid dose, like violence — they'll be things where you want to predict as well as possible. I think you'll want to add this part to it as well. Thank you.

Andrew Krystal:
Thank you very much. Our last panelist is Dr. Basit Chaudhry. Dr. Chaudhry is an internal medicine physician and medical technologist. His expertise is in the areas of health care payment, clinical service design, and the use of data analytics to improve clinical and financial performance in health care. He works to develop more scalable ways of delivering health care, founded Tuple Health in 2013. He's also been a lead research clinician at IBM Research. Please, proceed, Dr. Chaudhry.

Female Speaker:
You may be muted.

Basit Chaudry:
Sorry about that. I apologize. So, again, thank you very much for the opportunity to have this really interesting discussion. Just my background as a primary care physician, so a lot of my perspective on mental health is really from that context. Much of my work now involves working with high complexity patients, and it's very, very common that they've been overlapped between they're medical conditions and they're behavioral health issues. I think in many ways a distinction is very artificial, specifically given limitations and access to care. Many patients, around the country have two psychiatry's, specifically.

So, just in terms of kind of the vision — I'm just going to try to read some of this, just to try to stay on time. I kind of divided into kind of broad — kind of broad lines, then more specific, related to techniques from machine learning and narrow AIs. Like, kind of designate it here, and talk a little bit about challenges. So, kind of broadly, just in terms of a vision, from what I see for mental health and behavioral health issues, broadly [unintelligible] learning a system for mental health, in which generalizable causal inferences deepen and broaden across the clinical enterprise, in a continuous, repeatable, scalable, and cost-effective fashion. Here, clinical decision-making and process execution will be enriched by causal explanations, which progressively improve by feedback loops involving care delivery itself. These feedback loops will be operationalized through technologies that systematize through quasi experimental methods and extend them into the clinical enterprise, democratizing the rigor from research — the research setting into practice. I think one part — a major part of here, which I'm excited about is the way in which different kinds of insights can feedback into — and generate new theories and enhance the redevelopment. I think it's a major part of generalizing causal inference, obviously.

Broadly speaking, these hybrid care delivery and quasi-experimentation platforms will address threats to validity through two primary mechanisms. One is through structural design features. I think, in many ways, is the discussion in health care, and broadly — and slightly increasing the focus on data. I think, in many ways, we are not focusing enough on structural design features to address bias. How do we systematize those? I think, secondarily, [unintelligible], provide different quantitative techniques and modeling methods, through which you can minimize the reducible error inside of models, related to picking different kinds of functional forms — related to this within the context of, again, a narrow AI, so to speak. And I — my comments are outside of any kind of generalizable mentions of intelligence that can be [unintelligible] machines.

I think can, in theory, make meaningful advances across multiple fronts, particularly our counterfactuals, quasi-experiments, designing more relevant control/comparison groups, in rendering different kinds of functions — mathematical functions — for modeling purposes, that specifically change the variants-bias tradeoff that we're usually forced to make, much of which is just generated by advance in computational power.

The challenges to this, I think, are quite significant. I think one, broadly, right now, given that there's often not a definitive pathognomonic consolation of signs or symptoms for mental health issues. That creates a lot of complexity in terms of data quality and data annotation for supervised learning methods and related techniques. I think there's a very limited adoption of psychometrically validated instruments, particularly within the clinical enterprise. And that really diminishes the quality of data we have. Construct validity is, I think, an enormous challenge in this field, in particular, and something I get quite concerned about, because we often don't know exactly what we're modeling; broadly, data quality. Clinical data modeling, specific to mental health — issues of selection bias, omitted variable bias, all these things that we see when we work with existing data sets.

So, that's kind of, broadly — I think one other I just kind of mentioned, generating ground truth for algorithmic training is incredibly difficult. I think it's going to be especially difficult in mental health, both in terms of its cost and complexity. And, finally, I think I'll just mention, I think algorithmic interpretability — we really want to change the clinical inference. I think my fundamental assumption is that causal inference through explanation is critical to that. So, I think the interpretability of what we produce becomes especially important. So, I'll just stop there. Thank you. 01

Sarah Lisanby:
I just want to do a time check. We now have the remaining 15 minutes of this session for discussion to be moderated by Justin Baker and Andy Krystal, our co-moderators and also to remind the attendees you can insert your questions in the Q and A box, and Adam Thomas will be curating and bringing those questions to the panel. So, handing to back to Justin and Andy.

Andrew Krystal:
Thank you, Holly. We've heard about a number of different perspectives on what the future vision looked like — looks like — and it's clear that there are a lot of different components to this. I'd be interested to hear from the panelists about their perspectives on the other talks or presentations they heard and how they impact on their view of the big picture down the road.

Brian Shiner:
Hi, this is Brian Shiner. I talked about VA data. I was curious, a few of the presenters talked about deeper learning approaches than we're using. We're using basic tools.

[inaudible commentary]

Sarah Lisanby:
[inaudible] please mute your phones.

Brian Shiner:
Okay, I'm sorry. So, what I was just talking — so, we're using more basic ML techniques like, you know, random vars [spelled phonetically] and lasso and things that have been out there for a while, and, kind of, the challenge for us for thinking about those deeper techniques is, kind of, managing the data is such a task in itself and getting the data clean. Are you guys finding that you need to have less clean data for these more deep techniques or what — how are you managing, you know, spending all your time on ML when there's so much data management to do?

Sarah Lisanby:
Would one of the panelists like to respond to that or one of our —

Mi Hillefors:
Please feel free to un-mute yourself if you want to participate in the discussion because you're all probably muted at this point.

Basit Chaudhry:
I can — so, yeah — you know, Brian, on your point — just, like, I guess, from my experience, I think it's difficult to overestimate how important, essentially, the data management component of all this is. I wish, obviously, that weren't the case, but I think, if anything, we don't pay enough attention to it often. I think in areas that are very discrete, like computer vision and things like that where, you know, a lot of the focus has to — just by the nature of the field of the domain, you know, people have not had to focus on that as much in a certain way. And I think it's just very difficult within domain-specific applications, particularly ones which are knowledge intensive that data management and clinical data modeling I think just is a fundamental part of deriving, I think, better predictions and more importantly, I think, better causal imprints, which is what, I think, most directly patient care.

Neal Ryan:
If I could try a short answer to that one. This is Neal — whoops. Clearly, for panel data, lots of people are going through extreme boosting which may be a little bit better than the random forest but is similar — whatever — the deep learning stuff which gets, you know, a lot geekier is valuable for imaging data or image-like things or long-time series data or time-series like things and for language, and, you know, probably almost certainly would be enough better that it's worth doing, but it does get more complex by a whole lot and doesn't, you know, doesn't — none of it gets — makes the data cleaning and preparation a bit easier.

Brian Shiner:
Thanks, those are great answers, thank you.

Sarah Lisanby:
And Adam Thomas, you may want to bring forth some of our questions from the attendees.

Adam Thomas:
Yeah, we've got a couple questions that I'd encourage folks to type anymore questions they may have into the question box. The first one is focused on the data collection question that we need to be aware that the such useful data will need to be added to clinical patient meetings. Patients already have a problem with clinicians' split attention between them and the medical health record. Improving the medical health record is important, but is this problem on the radar? This comes from Reed Polo [spelled phonetically].

Sorry — I didn't decide who to direct that to. Andy, do you want to take that one first?

Andrew Krystal:
I'm sorry — were you directing that to Andy? Is that what you said?

Adam Thomas:
Yes —

Andrew Krystal:
Oh, okay —

Adam Thomas:
— or unless you want to re-direct it to another.

Andrew Krystal:
— well, is there anyone on the panel particularly inclined to answer that?

Tim Mariano:
This is Tim Mariano. I think I'll just say one thing very briefly. I, you know, I agree right? Adding burden not only to collection of the data, but also to the patient visits as well. I mean, I think the last thing that most clinicians will want is another is another dashboard to check and review, you know, either before or during the patient visit. So, you know, these tools, while great, also need to be minimally intrusive to the critical patient-clinician interaction, and whatever can be done on the back end to assistance that, I think, will be important.

Justin Baker:
Yeah, and this is just to — I'll just add to that, I think, you know, one of the things we think about as treaters is there's multiple purposes to a clinical encounter, you know, currently there's an information gathering component to that obviously, but there's also relationship-building component. And you asked, you know, what's the purpose of that relationship building, you know, partly, it's we know that the therapeutic alliance helps people by treatment and keep people sustained over longitudinal periods. And so, I think it's going to be really interesting to ask the question of, for the amount of money we're paying for particular bits of information, what is the actual cost to information quality relationship for all the types of encounters or bio-markers we could be capturing?

We heard this morning from, you know, using EEG, using, you know, trails from the EHR which, of course, come from those visits. We heard from — we'll be hearing, you know, from other types of biomarkers, like wearables. And, you know, you think about the information content of those things, they may be worth, so to speak, from that hour-long visit with the clinician, and yet, there's still some value to that relationship-building component, but it's not clear necessarily that we need to be thinking about the MD necessarily being the one in every case to maintain that relationship.

But I think we're going to have to, kind of, radically rethink models that we provide care in where we can really balance information into prognostic value against these, kind of, tried and true methods of patient-doctor relationship and really, kind of, strategize now in the access-high and resource-poor environment where we have far more need that we will ever have treaters available to spend minutes with people. So, I think there's — we're going to be talking more about that future state. Is it one where traditional psychiatrists providing that hour or where we have, sort of, networks of humans who can provide one-to-one patient contact and other methods to buttress, kind of, the information gathering piece?

Diana Clarke:
This is Diana —

Male Speaker:
Sorry —

Diana Clarke:
— sorry — this is Diana Clarke. So, one of the things that I'd also like to add is I really do believe that one of the things that we need to do is to bring the EHR company to the table as well, because if we can't try to integrate all of these things with the EHR, we will always have the clinicians and, or the patients moving from multiple platforms to get the information that we need. And these are valuable information to actually inform their treatment and patient care, but if we don't bring the EHR companies to the table and actually work on something so we can integrate these systems, we're going to always have this problem. And so, we will always only get a piece of the data to do anything that we do, and we'll always be missing pieces of information.

Sarah Lisanby:
I'd like to second Diana's point there. It's very important — and a way of integrating the information from data streams into clinical care at the patient encounter, and we're delighted to have a couple of representatives from one of the EHR companies with us today from Epic who will be participating in session three. All panelists across all sessions, please feel free to chime in during the discussion sessions if you'd like to make a comment.

Justin Baker:
Maybe it's worth it to go to [unintelligible] Doctor Laurie's question. [unintelligible] you want to just go ahead and un-mute yourself and ask the question about, kind of, ensuring that all of these new innovations are able to make it across the digital divide?

Sarah Lisanby:
Yeah, and so the community may not be able to un-mute themselves. So, we'll be —

Justin Baker:

Sarah Lisanby:
— reading questions from the Q and A box.

Justin Baker:
Okay, no worries. So, yeah, all those questions, is any thoughts on ways we can assure that the future innovations are available to underserved populations such that the digital divide doesn't become a digital mental health divide? And maybe I'll point that question at Amit.

Amit Etkin:
Yeah, so, thanks for the question. I think that's a really important issue of these. I think we often do look at things from the lens of, you know, research where we have a lot of resources and hospitals that do research tend to have a lot of resources and that's a major limitation. I think as we think about what the tools are, we also have to think about what the tools are providing. So, far, as we think about these digital mental health tools, there's yet to be really strong cases for what, you know, what they really add. However, let's just think of a canonical example of a smart phone-based tool. If it's something that becomes useful — becomes useful in part because it saves money for health systems, and that's really the root to which — root by which insurance companies and federal insurers will adopt things, because it ultimately leads to better care but saves money.

Part of that has to be thinking about the equipment and having accessibility is part of that. That has to be part of that financial equation. And I think as cheap as these things are becoming, it's now becoming quite feasible to provide smartphones. There are programs for elderly and low-income people, for example, giving them smartphones with data plans. So, I think that those things are overcome-able, we just have to think of that as part of the package exactly as you raised it.

Justin Baker:
And I think there's no question that at this time, even in the past two weeks, people who didn't have a cellphone or a tablet or some access to the internet, there's strong consideration of multiple levels to get all those new technologies into both conventional care settings, even inpatient units where cameras and other things haven't — because of privacy concerns — been the preferred mode. But now, that's being considered which is exciting from the standpoint of accelerating the move towards telehealth type solutions that would work if all you have is a device. It's also rapidly raising some of the ethical considerations. So, excited to hear more about that later today. We have about four more minutes —

Diana Clarke:
I was going to just say something really quickly — Diana Clarke. Yeah, this is really, very, very important to me, and one of the studies that we're currently doing through the APM mental health registry, which is a quality measure development project. And one of the things that we do find is that a lot of the — a number of patients don't have access to the internet. So, they're not able to participate. But we also have to remember our patients as well and realize that their level of comfort with technology is also a barrier that we're going to have to overcome. And so, how do we reach out to patients — mental health patients — speak to them in language that they can understand so they can feel comfortable using it?

So, it's not having access to the internet, but it's also patient's basically saying, “I don't do the internet. I don't complete anything that has any access to anything.” And so, how do we overcome that barriers so that the information and the data that we're collected [sic] is really generalizable to all the patients — all patients with mental health issues — mental and substance use issues?

Andrew Krystal:
I just have a quick question for the panel. The presentations have all described a number of different types of data and that are likely to be important in the future vision and from lots of different sources. I'm interested to hear what people's vision is about how integration could happen? Did people see the — let's say, the NIH having a role in this as somehow being a clearing house for data and predictive algorithms? Appreciate your thoughts on this.

Justin Baker:
Yeah, I'll say just maybe open it up since I hear we have just a couple minutes left, but to ask, sort of, more broadly, what's going to be the role of industry versus NIH in, sort of, maintaining those large data sets because I think there's, you know, there's lots of companies whose job it is to maintain large data sets. It's not traditionally the domain of health care, although I'd be curious what panelists think about maybe in five years or less, what is going to be the distribution of responsibility around these issues?

Sarah Lisanby:
If you're speaking, you may be muted. I was wondering if Basit might have a view on that question.

Basit Chaudhry:
— mute — sorry, I had to change locations. But yeah, I think if you look at, for instance, some of the, kind of, different techniques we've talked about from machine learning and different forms of quantitative techniques, like things, like, deep learning, you know, the advances made in those fields have been driven largely by two things. It's interesting. A lot of the algorithmic work in that is — in those areas — is quite old. I mean, dating back to, like the 1980's. The advances that recently made that a lot of people do the kind of sophisticated, say, natural language processing that we talked about before really falls in two things.

So, one, it's the implementation of computationally intensive graphics chips, and then, two, large corpuses of well-annotated data. So, for instance, in computer vision, one of the major reasons why so much progress has been made is the data available through ImageNet. And so, without high-quality corpuses of data that are well-annotated, like, I can't focus — or emphasize that enough. It is very difficult to train algorithms in sophisticated ways for domain-specific applications. So, again, a lot of the innovation we see now is literally related to data available through ImageNet that's been annotated by humans.

Sarah Lisanby:
So, —

Justin Baker:
Wrap up quickly — is that okay —

Sarah Lisanby:
— yes, go ahead.

Justin Baker:
— thank you, Basit. And I completely agree that annotation is really one of the next key roles for health care in terms of thinking about the of [unintelligible] going forward. If you can't measure it, you can't manage it. And we're seeing maybe an influx of a measurement culture in psychiatry which I think is a really good thing, and I think it's been present in oncology and every other branch of medicine. It think what we're all talking about today is this new contrary of data types and willingness to measure things in psychiatry that hopefully in five, ten years, or maybe sooner, will allow those learning systems to begin to churn on that data and iterated and develop novel therapeutics.

So, thanks, everyone, for providing their exciting visions for the future of the field. Thank you also to my co-moderator, Andy Krystal, and all the panelists for taking the time as well as everyone listening. So, I'll turn it back over to Holly who can take us to the next panel.

Sarah Lisanby:
Thank you very much, and I think that's a good segue. We've — now that we've talked about where we'd like to go to achieve a health system that's able to provide effective mental health care for all patients wherever they receive their care and one that takes into account the social determinants of health. The next section, we're going to look at where we are now. What can we learn from existing networks, and I'd like to introduce our moderators for this session; Guillermo Sapiro, is the Pratt School professor, engineering at Duke University, and he works on topics of theory and applications in computer vision, computer graphics, medical imaging, and machine learning.

I’d also like to welcome Dr. Helen Egger, who is chair of the Department of Child and Adolescent Psychiatry in YU, and an innovative researcher employing new technologies to advance mental health care, especially for integrative pediatric mental health. Dr. Sapiro, please take it away.

Guillermo Sapiro:
Thank you very much, Holly, and thanks everybody who are participating. As you can see, we have a very prestigious team of speakers and discussants coming up in the next 45 minutes. We have five minutes for each speaker, that will take us about 30 minutes for the first part, and then 50 minutes for discussions. [inaudible] you see, every speaker, which takes a bit of time. We'll let them use all the time. You can see they're very prestigious speakers that we have now basically on this slide. They're very diverse and each one of them is a renowned expert in the field, and they come from multiple institutions. So, we are — as I say — we are not going to use up their time to present their bio. So, we can give them the time to keep us and to educate us.
We also came with a few guiding questions with Helen that Helen is going to present next before we move to the speakers. Thank you and stay safe.

Helen Egger:
Thank you, Guillermo, and thank you to Holly and the rest of the organizers for including us in this wonderful conference. We had some questions that we wanted people to think about as they were listening to each of the presentations, and many of them have already come up in the first session. So, we put them in, sort of, three buckets. The first is around these questions about data acquisition and curation or the structural design features and data management; so, where does the data come from, how is it measured, who are the subjects, and importantly, I'm speaking as an epidemiologist, what is the sampling frame of the subjects who participate?
Then, issues around data combination, integration, and also was brought up, data management. I think the question of how we combine multi-modal data, in particularly over time, is a key challenge, how we deal with missing data. And then, how do we test the clinical significance of our results in the population we created it for, but then in other use cases, and particularly, it came up before as an interest of mine around health disparity populations. Then lastly, the huge challenge of bias in — which comes in many ways. We give one example here about how often the mental health information is incomplete in the EHR, and there are many other examples. And then, always the pesky question of how we distinguish between correlation and causality.
So, that's the frame, and now, I'm going to hand it back to Guillermo to start our speakers.

Guillermo Sapiro:
Thank you very much, Helen. And we — as we say, we have five minutes per speaker. So, Aziz, you are first, and we're — hopefully, we'll not have to give you the five minutes warning because you're [unintelligible] — I apologize in advance if we do. Thanks.

Aziz Nazha:
Thank you very much. So, thank you Dr. Lisanby, Dr. Gordon, and thanks for all of you for the opportunity to be here with you in such a great workshop. So, I'm an oncologist. I treat cancer patients. My specialty is treating a blood form — a form of blood cancer, Myelodysplatic Syndromes, and I also direct the center for AI. So, we do some — use AI and machinery in deep learning to try to solve some of the challenges that we have clinically.

So, just to give you a quick background about the disease. This is a form of blood cancer that happened in the blood [unintelligible] diagnosis about 10 [unintelligible]. When we make the diagnosis, actually by doing the bone marrow biopsy and try to put the – an experienced amount of pathologists can look at the slide and give us the answer whether the patient has Myelodysplatic Syndrome or another form of blood cancers. And then the next step of finding cancer is to, kind of, stage the disease, and this is, kind of, a blood cancer so we don't do CT scans. What we do — we just take the blood count, look at the patient's chromosome, block them into scoring system, and then come up with the risk score [spelled phonetically] that this patient is a higher risk of progression to advance stages and lower risk.

And this is a very important step because our treatment guidelines are based on risk. So, if patients has a higher risk, we tend to give them more aggressive therapy compared to a patient with lower risk. So, what we found with — you see on the slide — several of the challenges that we have clinically, and then the question can resolve these challenges using machine learning and deep learning. So, one of the biggest challenges is the diagnosis of the disease. It is very subjective. So, about a third of the slide — of the bone marrow slides — come to us from the community oncologist. They have misdiagnosed the disease, whether they thought the patient may have it, but they don't, and vice versa.

So, what we have done so far, we took some of the pathology slides and using computer vision, we've been trying to say, is this MDS versus not MDS? And we have very exciting results that's not shared yet. But the other project that you can see on the lower slide is looking at the blood count and the genomic signature from the patient, can rebuild the machine learning model that can differentiate MDS from other minor malignancies? We have done this project where we actually compared MDS to other — what you see here — other minor malignancies look exactly like MDS. What we're able to find, actually, we build machine learning model that achieved about [unintelligible] under a curve of 96 percent, differentiating MDS from other disease just using clinical variables and mutation data. So, you could envision a future of not doing a bone marrow biopsy.

The other thing, that in terms of prognosis, what we found that the — what we are predicting in terms of this staging system, what the actual survival of the patient is completely different. So, we have used machine learning to take clinical data and the mutational data and built this model that can provide personalized prediction. And if you can select here — this is supposed to be a video — if you can click on the — actually play, you can see me here changing some of the clinical variables of the patient — make the patient actually worse, and then you can see this, sort of, the viable probability of the patient, it changed.

So, this is just to talk about, you know, personalized prediction and data specific for a given patient. And finally, they're used also some machine learning in technologies trying to predict response to chemotherapy. So, one of the chemotherapy that we give to our patients is called hypomethylating agents. Today, we don't have any models to give us an idea which patient would respond or not, and it takes about six months of patients to achieve a response. So, one of the projects we did, we took actually the recommended system algorithm, saying if you watch this movie, this movie, this movie, you're going to watch this movie, you [unintelligible] you're going to want — you're going to respond or not respond. Based on that algorithm, we developed genomic biomarkers that can predict resistance in about a quarter of the patients, but when they present, their accuracy of prediction of resistance is about 93 percent on an independent validation cohort.

This was also validated in the lab where we used the CRISPR task technology to knock out those genes and we generated those cells in the lab. And finally, one of the projects were working on now is using time series analysis. So, sequence in sentence matter. When we give chemo, the blood counts change over time. We monitor the change in the counts over time, and we predict response or no response. We were able to build a model that actually just monitoring the changes in the blood counts on the CBC. We were able to predict the patient was going to respond to chemo or not with 83 percent area under the curve. So, these are some of the examples that we have used different type of technologies and machine learning and deep learning and different domains trying to improve our tools knowledge and clinical decision-making tools that help our MDS patients.

Guillermo Sapiro:
[inaudible] Aziz and thank you all for keeping the time. The next presenters are going to come in tandem is Greg Simon from Kaiser Permanente and Rebecca Rossom from HealthPartners. So, they're going to take it one after the other, 10 minutes and — hey, Greg, please go ahead. Thank you very much.

Greg Simon:
Actually, Rebecca's taking this first part.

Guillermo Sapiro:
Oh, okay. So, then — perfect — thank you. Thanks — it's your 10 minutes, your — you two together will be within the next 10 minutes —

Rebecca Rossom:
[laughs] Okay.

Guillermo Sapiro:
— [inaudible] —

Greg Simon:

Guillermo Sapiro:
Thank you.

Rebecca Rossom:
We'll do that. So, this is Rebecca. Greg and I are from the mental health research network, which is a consortium of 14 research centers embedded in large comprehensive health systems. Together, we serve a population of about 25,000,000 people across 16 states. And, sort of, related to the conversation earlier, we all do use Epic for our electronic health records. So, we each maintain a data warehouse following the Sentinel, PCORnet common data model and consolidate our data that includes electronic health record data, insurance claims, pharmacy dispensings, patient reported outcomes, lab results, demographic characteristics, insurance covering characteristics, and mortality when we link up with state mortality records.

So, using these data sets then, we are — have worked — done some work to develop suicide risk prediction and really have worked to predict suicide risk accurately and relevant to the point of care that we can estimate risk for a specific customer. So, we have developed validated models predicting suicide attempt and suicide death after mental health specialty or primary care visits, and now, we're working on developing models for ED visits for patients after inpatient stays. Next slide, please.

So, for this model, we used 25,000,000 visits for 4,000,000 patients and developed models that have an overall accuracy or area under the curve of 84-86 percent. So, that's what we have. What we'd like to see is more accurate coding of self-harm events to the discussion previously. You know, we recognize the models are as good as the data we can put in it, and the more we can minimize those false positives and false negatives, the better models that we'll have.

We'd also love to have better up-to-date social determinants of health, so things that aren't routinely collected in the EHR or are not routinely updated but can have significant prediction for suicide attempts; things like job status, relationship status, food and housing security, for example. What we don't think we need is more complex or opaque modeling strategies. You know, we have an evaluated alternative modeling approaches, including classification or tree-based approaches, classification and regression trees, mixed effect regression trees, random forest plots, but honestly, so far, we found that the simpler models are nearly as good. We also don't think we need new model development for every healthcare system, and we're not letting that — the perfect be the enemy of the good.

In our care systems — mine and Greg's — our suicide risk model has already been implemented and that happened within about six months of the publication. In my healthcare system, by the behavioral health case managers in our health insurance are in Minneapolis and within about nine months in a pilot behavioral health clinic at Kaiser Washington in Seattle. Next slide, please. Additionally, a tribally-owned and operated healthcare system serving over 65,000 Alaskan Native people in Anchorage in 55 real villages took our model predicting suicide attempt after a primary care visit and re-validated it in a sample of about 50,000 visits in their clinics, and this table shows results from that re-validation. And you'll see that the positive predictive value is a bit higher reflecting a higher prevalence of high-risk patients while the sensitivity was a bit lower, overall suggesting good usability in a very different system. And this healthcare system is now also developing an implementation plan. So, I'll turn it over to Greg.

Greg Simon:
Thanks so much. Could you go on to the next slide? So, I'm going to shift a bit and talk about some of the work we're beginning regarding developing personalized treatment pathways for depression, and by way of introduction, I should say that Roy Curlis [spelled phonetically] and I — literally 10 years ago — wrote a paper about what we needed to develop personalized treatment pathways for depression, making the point that we thought traditional randomized trials were not getting us there and would not likely get us there, and we said what we really need are comprehensive data on treatments and outcomes coming likely from large healthcare systems where we could have the data volume necessary to answer some of these questions.

If someone had told me 10 years ago, “Greg, 10 years from now, you will have made almost no progress on personalized depression treatment, but you will have accurate predictions of suicidal behavior,” I would have laughed out loud. I would have thought that the likelihood of progress in those areas was exactly the opposite, but that's where we are today. So, a few words about, sort of, you know, maybe why it's been a bit slower getting started on the personalized depression treatment.

So, what we really want are — oops — back to the previous slide — okay. So, what we want. We would like patients and clinicians who are selecting treatment for depression to have accurate and constantly updated personalized predictions, essentially to say for someone with your characteristics, those characteristics being your own patterns of symptoms, patterns of co-occurring conditions, your history of response to various treatments, and even other, sort of, omit data that might soon be available in electronic health records. This would be the next step for you, and I'd say where we are now is we're just getting started. So, maybe you could go one the next slide now.

So, in terms of what we have, you know, just looking at, say, six of our health systems that have, I'd say, made the most progress in terms of systematic collection of outcome data. We have a database that includes about 1.4 million people who've received some form of therapy for depression just in one year, in 2019, where we have comprehensive data about their exposure to medication treatments for depression, and this is, you know, come online within the last two to three years, relatively complete data regarding outcomes. So, the database regarding outcomes comes in those six healthcare systems includes longitudinal data about 3.6 million PHQ-9 depression score observations for about 1.1 million people.

What we find in these healthcare systems is people initiating treatment typically have a standard, typically the PHQ-9 depression score recorded about 85 percent of the time and have outcome scores recorded typically 60 plus percent of the time. The failure to collect outcome scores is, as many of you know, not because people are seen and outcome is not assessed, but because people drop out of treatment.

So, what do we really need to move forward on this? Interestingly, we have very good data on outcomes. We have really poor data on adverse effects of treatments. We may know that medications are discontinued and even when we try to mine text data to find out why and what particular adverse events people experience, they're poorly recorded. Although we can characterize pharmical [spelled phonetically] therapy fairly accurately, all we can say is psychotherapy is that a visit was occurred — visit occurred — and, to be honest with you, a bill was paid. We can say very little about the actual content.

We would really like to have more useful data regarding the logic of treatment decisions, that may sometimes be recorded in free text, but to understand when a new medication was prescribed, was it prescribed because of adverse effects? Was it prescribed because of lack of effectiveness of a previous treatment? We'd really like to see that recorded in s systematic way.

And then, the analytic challenge here, I think, is different from, you know, the challenge of suicide risk prediction where we really can use, I think, fairly standard and well-established methods for a “yes, no, discrete outcome” and a fairly well characterized vector of features or predictors. Here, we're really trying to discover treatment response phenotypes. We're trying to understand what patterns actually exist in nature of responses to different treatments. The number of patterns is quite large especially when we consider the possibility order effects, in what order people were exposed to different treatments may matter or predict in terms of subsequent outcomes. And we really are moving just from prediction into causal inference.

So, we're having to combine a, sort of, pattern discovery activity with a causal inference activity. That gets fairly complicated, and that's something we're just starting to think about. We do have a couple people in our network who are working on these methods, you know, quite actively right now. So, that's my sense of, you know, maybe why we're not quite as far along in the personalized treatment, but what we're hoping to develop. I'll stop there. Thanks.

Guillermo Sapiro:
Thank you very much, both Rebecca and Greg. And we are going to move to the other side of the ocean to Matthew Hotopf to talk —

Matthew Hotopf:
— and the idea of this is really to try and get a handle on people's 360-degree view of people who experience depression and try to predict depression outcomes using mobile technologies. If you look at the diagram at the bottom of the slide, the current state of play in most health systems and most research is that we get snapshots. We get snapshots of people over the course of a long period of illness — depression is a chronic disorder — it relapses and remits — and we simply have these little snapshots in time, usually only when the person's got a diagnosed disorder not before and usually those snapshots are entirely dependent on them being unwell and seeking treatment.

In, you know, the challenge for mobile health is to be able to provide a continuous, pervasive stream of passive data as well as some active data which can give a very well-rounded indication of many of the higher functions of the central nervous system including sleep activity, sociability, cognition, speech, and mood. And we do this using a platform we've developed in RADAR-CNS which both takes passive data from a smartphone, so it'll include things like GPS. Also, wrist-worn wearables, in fact we're using a Fitbit in both this study to get actigraphy and also, active tasks, questionnaires, little bouts of sampling, and cognitive and speech tasks.
The RADAR — so, RADAR's actual active in three diseases. We're doing this in epilepsy and MS as well as in depression.

But for the depression study, we're just about at the brink of having reached our recruitment target of 600 participants followed for up to three year over this time with data coming from Android smartphones and Fitbits with intermittent web-based interviews and also, more detail across a, sort of, qualitative interviews on their experience of the study. We — so, the aim of the study is really to try and identify bio-signatures which are indicative of relapse. But, it's a very naturalistic study. It's a study which is taking people of all different stages of depressive relapse and remission and following them over time in as unobtrusive way as possible. Next slide, please.

So, next slide, please — yeah, great. Okay, so, I was asked to, sort of, highlight some of the questions and issues and how we've tackled them. We've been running — I guess we've been data collecting for about two years or so, and the first question — I think it's really important actually in this whole seminar — I haven't heard enough of this, about the patient — the participant — and their experience because all these technologies aren't profoundly — they require a sent or consent for data to be used. They have indications around privacy.
In the RADAR-CNS program, we're asking people to give us GPS data, which is obviously highly personalized. We do that by obfuscating it, but still, the issues around privacy and acceptability are critical. So, we have had a really serious, intensive periods [sic], and it's ongoing in the study of public inpatient involvements and a very active patient advisory board. We have a lot of qualitative work. We did a very, sort of, detailed workshop with patients and participants about what they would tolerate in terms of device selection.

So, we're asking them to wear a device for two or three years — what matters to them and what matters to them might be quite different to what matters to researchers. So, in fact, we went to a consumer-graded device rather than a research device because we wanted to make surer that participants had something which didn't feel stigmatized, that was easy to use, the batteries — the battery life was reasonable, and so one, whereas the research-grade devices where we might have been able to get raw data off the device, had more difficulties in those respects.

There's also a need for technical support, and we — and I actually, in the mobile health field, certainly when we were starting, it was actually quite difficult to find a platform which did the job for us. What we wanted was something which was device diagnostic so that you could plug and play new devices into it. It wasn't, sort of, caught up with a particular manufacturer's device. And we built this — we also, I think, worked — made the decision to make it open source. By doing that, you actually make it far more sustainable. It doesn't just last for the length of the project. It actually has a life beyond that.

So, there is — if you want to look it up — a platform called Radar Base, which is used to being, sort of, the study. There's also — I put there, technical stuff happens because I actually — when you're doing this, you're not in control of a lot of technology. During this time, Android changed their operating system. It stopped us from being able to do some things which we were taking for granted that we could do. Suddenly, notifications weren't getting out to our participants. And you have to have a very, kind of, nimble way of identifying problems and fixing them in the middle of a study in a way which most studies are wrote on the plans and protocols of — its just not something you normally experience.

Another issue is future adoption. So, we want this to get out into the real world if we demonstrate something beneficial, something which is actually likely to be useful. And having early engagement with regulators as being really important. So, we have had conversations with FDA and EMA about how this would work out. But in terms of its use utility and drug development would also be on that in terms of something which could be integrated into health systems.

And finally, data analytics, and this is work in progress. I am an epidemiologist who's used to doing survey data. I've never had data the scale of this, the change of this, and, you know, the varying temporal resolutions and the scale of data is absolutely enormous. So, these are huge challenges, in particularly the issue of data missingness because participants will drop out of this study and draw back in during the course of the treatment. That data missingness may often be actually quite informative. It may be the first thing you do as you start getting depressed is you stop answering the questionnaires which we're posting out. Thank you very much.

Guillermo Sapiro:
Thank you very much, Matthew. And we are coming back to this side of the ocean with Peter. Peter, please go ahead.

Peter Zandi:
Thank you for inviting me to come and present. I'm presenting for the National Network of Depression Centers and our effort to establish a learning health system for mood outcomes — for mood disorders, bipolar disorder and depression. You know, the last five years, we've been working on building the infrastructure for this learning health system. So, we're not quite there at the point where we're starting to analyze data and apply some of the methods that we're seeing here, but we're hopefully getting close to being at that position. And so, what I want to provide is a high-level description of the stuff that we've been working on.

Really, the NNDC is a consortium of around 26 academic centers around the country that provide care for patients with bipolar disorder and depression. The goals of the consortium are to advance care — patient care — for these patients to promote research and to promote education around the mood disorders. So, we've been working on establishing this mood outcome program, which is our flagship initiative for trying to create this learning health system for mood disorders across the country — across these 26 academic centers.

The Mood Outcomes Program — there's really four goals — for the Mood Outcomes Program. The first really is to implement this idea of a measurement-based care program as standard of care across all our centers. There's a lot of evidence that measurement-based care works and is used for patients, but there's reluctance to adopt —

— reporting requirements from CMS, like MACRA. The third goal of the Mood Outcomes Program really is to facilitate this collaborative multisite, prospective research in these real-world settings across these academic centers. And the fourth goal really is to be able to take those findings from these efforts and be able to translate them back [inaudible] care for our patients across each of these centers that are participating in the consortium.

So, in this way, the idea is that this Mood Outcomes Program can serve as an engine or driving the adoption of this learning Health System across each of the centers that are participating in it. So, he's been working on this initiative for the past four or five years to try to develop the infrastructure to support this program and to help us to get to that goal, and we've made considerable progress over those four years. And, you know, the first step was to agree on a standard set of measures — patient-reported outcome measures — that we wanted to capture from our patients at every clinic visit, so that it was, like, capturing vital signs from these patients as they are receiving care in our centers, and that the clinicians would use that information as part of improving care.

So, we went through a process of working with each of the centers, and we agreed on pre-measures that capture at every visit the PHQ-9 which we've heard about from other centers, the GAD-7, and the Columbia Suicide Self-Rating Scale. And then, we developed a IT infrastructure for patients to complete these measures across each of the centers and then for the results of those scales to be viewed in real-time by the clinicians, really be used for measurement-based care, not only for screening, but for continual tracking of the patients as they are providing care to the patients.

So, we've gone through a pilot phase where we've implemented this program at eight centers across the country. We've enrolled 10,000 patients, and we just finished that pilot program, and we're very encouraged about it. And we're now at the step — we're going to be trying to disseminate this program the rest of the 26 centers across the consortium so that we we're disseminating it more widely. And we also have been involved in an initiative working with Epic.

Over two-thirds of the centers are Epic — use Epic — as their local electronic medical record so that we can pull other data from the electronic medical record including counters, problems, diagnoses, medication, procedures, and we can pull that data with the patient-reported outcome measures — pull it into a central repository where we can then use that as a platform for the types of work at we want to do, whether it is quality improvement or these large-scale, multi-site research projects where it's analyzing secondary analyses, exiting data, or to embed studies in these real world settings whether it's a comparative effectiveness study or a pragmatic clinical trial. So, we're in the process —

Guillermo Sapiro:
Sorry, this is Guillermo and I truly apologize to Peter and to Scott that is coming next, but I just got a call from NIH saying that we're, like, five minutes behind. So, if — I apologize, but if we could cut a few minutes from each appeal, I apologize.

Peter Zandi:
Sure, sure. So, I'll finish real quickly — sorry for going long. One thing that I think is interesting as we've leveraged this program that we've established just recently to engage and initiate a large-scale study of ECT across our centers. We've implementing — we're implementing this program in our ECT centers across these academic centers that are participating in the program. We're harmonizing how we're documenting ECT, and what we're going to be doing is as our patients are receiving care through ECT, we're going to be collecting biological sample so that we can carry out genetic studies, look for genetic markers, severe treatment-resistant depression, and response to ECT both in terms of efficacy and cognitive side effects.

And our hope is that we'll be able to use that knowledge that we gain from that large-scale study which we're doing in collaboration with the PGC, Psychiatric Genomics Consortium, to identify markers of who are good candidates for ECT and who might respond well that would then feed into these, sort of, machine learning and AI algorithms that we're talking about. So, that's just the beginning of the sorts of projects that we think will be really exciting that will be able to build on top of this program.

The next slide laid out a couple of challenges. But I think that they've all been, sort of, talked about in one way or another from other folks. But I think, in the interest, Guillermo, of the interest of time, maybe I should stop there and pass it on to the next person.

Guillermo Sapiro:
Thank you very much, and again, I apologize. We could hear and listen to each one of you guys for an entire day, but we have a busy agenda. Scott, you're next. Thank you very much.

Scott Kollins:
Yeah, yeah. Can you hear me okay, Guillermo? I know there's —

Guillermo Sapiro:

Scott Kollins:
— great. Well, thanks. I think I should be able to keep this brief, you know, and I want to talk about a program of work that is relatively early days, and so, I don't have as much in terms of data and results to present. But, just to talk about a couple of, sort of, main problems or questions that we're working on. I'm a clinical psychologist and professor in the department of Psychiatry at Duke, and I'm also the director of the Duke ADHD program. And we have been, sort of, thinking about this for a couple years along with myself — along with Holly — including Guillermo as well as Jerry Dawson and others here at Duke.

One of the things that we've recognized is that child health and a child mental health are, sort of, relatively understudied and under-addressed in terms of the applications and data science to improving these conditions, and being trained as a child psychologist, this is, sort of, a nice place for me to start thinking about how to use some of these methods to improve the lives of kids. So, we're working on a few things. This is funded in part by support from NIMH, as well as an ICHD and some internal support [inaudible] kind of, two basic questions.

First of all, can we improve the way that we surveil risk for both ADHD and autism as well as other more general neurodevelopmental problems using primarily EHR-based risk prediction models and taking information that is available only through the EHR applying machine learning, natural language processing, and being able to identify kids earlier than they would otherwise be presenting for an assessment or a diagnosis of autism and ADHD? And then, the second part of this work is prospective, is to validate the models that we are able to build and then see if there's additional information that we can gather that would improve or refine the models by performance and that includes both sophisticated computer vision-based tools that Guillermo and others are working on as well as the addition of other patient-reported outcomes and surveys and things like that.

Part of the platform to do this prospective validation through an Autism Center of Excellence that Jerry Dawson and myself are PI's of. One of the projects in that Center is we are ascertaining all kids that come through Duke Peds Primary Care starting at 18 months, and we're able to track them, see who ends up developing both ADHD and autism, and then use additional questions that are part of the study, additional surveys and so forth that are part of the study, to see if that improves model performance. You can go to the next slide, please.

So, you were also asked to talk a little bit about successes and challenges so far. So, in terms of our successes to date, and again, this is relatively early days — we've been working on this for about a year to 18 months. We've had had both institutional as well as NIH funding to support this work which we are grateful for, and we've been able to successfully extract and curate a fairly large dataset from the Duke Health System, about 30,000 patients, and started to work on the modeling using the EHR. And so today, we've got — we actually have a decent baseline model that wear continuing working on to refine, and we've already got a — we've got a paper under review now, one of our fellows that is showing significant differences between kids who are later diagnosed with autism or ADHD in terms of their healthcare utilization as measured through the health record.

We've had some challenges, and I — this isn't any surprise to this audience — the complexity of the EHR data is profound, and we've spent a lot of time working through that. We know that a lot of the important risk factors that are in the health record or part of the mother’s health record and even the father's, and that has been a challenge that we're working through to be able to link those data to refine the model performance. And one of the big ones that we're starting to really think about and, you know, having heard some of the other presenters, it's exciting to hear the other, sort of, precedence in this area, but for us to think about, what do we do when we get a good model and how are we going to integrate that into primary care.
We're just now starting work to understand different stakeholders’ input on how they would use or think about risk if it's identified early. So, what do you do with a model that says you've got an 80 percent chance that your 12-month-old will later be diagnosed with autism and ADHD? There's a lot to understand about how primary care pediatricians as well as patient caregivers — how they're going to be using that information. And so, we're really interested to start to understand that better so then we can start to develop interventions and deploy them through the health record and other places. So, I think I'll stop there. Thank you.

Guillermo Sapiro:
Thank you very much. Thanks to all the speakers, and I will pass this to Helen to moderate the discussion. Thank you everybody.

Sarah Lisanby:
Helen, you may be muted.

Helen Egger:
I now am un-muted, thank you for the reminder. You think we do this every day that we would remember that. But thank you so much for those great discussions, and now, we actually have four discussants who are going to make some comments on exciting projects that were presented. One of the things someone said about patient engagement and burden — one thing I haven't heard that I want us to be thinking about is the importance of human-centered design, and not only, sort of, the content of our tools, but how we present them and that those are going to impact not only our results, but also our ability to scale our tools to be used in actual practices. So, now, I'm going to turn to Laura Germine, it was from McClain.

Laura Germine:
Can you hear me?

Helen Egger:
Yes, we can, yes.

Laura Germine
Great. Let me know if I start to fade out. So, I absolutely agree, you know, to the discussion about patient-centered design and thinking about what the role of the patient is in patient programs. I think a critical piece that we often have as a footnote or a future direction or any piece in all the discussions about clinical applications of machine learning and AI, but I think that, you know, one thing that is, I think, a challenge for the field is how do we put that in the middle of the project? How do we think about not just, you know, talking to patients in the form of, you know, patient [inaudible] which I think is really critical and really important.

But, you know, how can we engage patient communities to actually be a part of the process of developing algorithms? So, you think about how many really smart data scientists and engineers there are out there who are part of — maybe they're patients themselves, maybe they have family members who are patients. One of the things that's been striking to me about this situation we find ourselves in with COVID-19, the pandemic, is how much people who are not disease experts have actually contributed to our thinking about prediction about what's going to happen, and what are the right public health interventions, you know. Seeing the collaboration between epidemiologist and everyday people who have good skills in these areas has been really remarkable.

So, how do we think about engaging patient communities in the process of algorithm development is one point. And the second point that's related is, you know, where does open data fit in here? So, we worry about privacy, but what about sharing? Are there conflicts between the incentive structure around IP and around, you know, future commercialization? And this incentive structures of science that are really preventing us from having data be shared in a way that would be maximally — would advance scientific progress. So, those are just, I think, the two points that I would like to have us talk a little bit about, at least bring it to the discussion.

Helen Egger:
Those are fantastic, thank you. I think we're going to go through each of our discussions to give these short responses and then address questions that are coming through. Roy Perlis from Partners, can you go next, please?

Roy Perlis:
Sure. So, I really — I enjoyed all the presentations. It's nice to hear everybody pulled together in one place. So, a few years ago, I was presenting some of our natural language processing or doc hocus pocus at Broad and one of my colleagues leaned forward and said, “So, is this a problem of science or engineering?” And I think that's a critical point for us to keep in mind is that much of what we're talking about it's not that the science isn't there, but it's really a question of engineering in the sense of, you know, we have as Greg and others mentioned, we have models that by objective performance metrics look very good, and so, I do think one of the things we have to consider is why has there not been more dissemination?

We and others have been doing this kind of thing. I published a machine learning for treatment resistance prediction model eight years ago, not replicated, not used, and so I think one of the big challenges that we should be explicit about that people may not be comfortable acknowledging is that Epic is the 800-pound gorilla and Epic is a massive barrier to progress and dissemination. So, it's wonderful that we all have — many of us have access to this sort of common data model, but it's also a huge challenge when Epic owns, by contract, any code that touches theirs, and it's just very hard to build on top of it.

I want to mention a few sorts of specific things. So, the genomic network had an RFA out for machine learning papers, and so I've probably seen over the last year a hundred plus psychiatry, machine learning clinical application papers. I wanted to just highlight three things we need to talk about besides sort of engineering. One of them is, I don't care about your AUC, and I don't care that your AUC [unintelligible] data too, especially for rare events. And so, I think it'd be really helpful for the field if we focus on PPV and NPV and another sort of clinically interpretable parameters, as well as calibration, right, so how well can I stratify risk?

I think we need real [unintelligible] validation. What does your model look like in another health system? So, our standard has become, you know, cross-validation is great, but how does it do across the country or across town, and that I think leads into the final point, which is we have to get past this sort of Groundhog Day approach to machine learning where everyone presents their model as if there were no prior art.

You know, again, we're now five to eight years into this era when lots of us are building models, and yet it's always the first-time other models, so you know, significantly exceeds chance. And so I think the last point I would make is, first of all, the default should be if you're presenting a new model you don't just compare it to chance you compare it to the one that the folks down the street or across the country built and I'm still not seeing that for things like suicide prediction. And also, we have to compare it to actual clinical practice and sort of what are [unintelligible] not just chance because we hope clinicians do a little bit better than chance, so we need to think a little bit about how we evaluate our models. So, those are some challenges for the very smart folks on this call I think to consider in, how to get to dissemination.

Helen Egger:
Roy, thank you so much. Those were excellent and embracing comments that I think really frame the things we've heard and will be hearing. I'm going to turn now to Tanzeen Chowdhury from Cornell.

Tanzeen Choudhury:
Thank you. So, I wanted to kind of make three points and one is around the kind of patient-centered or user-centered design. I think we have a broken model of chasing after an engagement. We talk about how do we engage patients, how do we engage doctors in a lot of the measurements that we can collect. Just like we — even if we had a system where a doctor came to our house to deliver treatment, we don't want them to come every day, right, that would drive us insane. We think of engagement that's something that the user should check every day or multiple times a day. I think it should be much more geared towards where the need is. When I'm doing well, everything's going well, the system should fade into the background, it should engage and deliver me useful content and actionable content when I need it. I think we are kind of driven by the model of Twitter and Facebook are that you have to be always on and which I think is broken and we haven't addressed adequately enough.

The other is kind of building upon Roy's points is that what cross dataset and insights can we get? I think we have all these results that show predictive ability, but nothing that tries to bring datasets together, and I think that's something NIH could incentivize with the mobile [spelled phonetically]. Now there are multiple centers that have collected [unintelligible], I work a lot on the mobile sensing data, but others have worked on online data, but how do we bring them together and make sure they work, and I was — and I'm interested in the data that Rebecca presented in the suicide risk data but [unintelligible] and some are reproducibility in Alaska. I think we need a lot more of that.

And the other thing I would say is, if we think about measuring technologies in the technology realm, we have a lot of measurement. Right now, we can say in the mobile context we can reliably measure a lot of things without user burden and all sorts of things and then we have a huge slew of intervention that works on mobile devices continuously for the patient, but they are not connected, right. Yes, you go and say you need eight hours [unintelligible] 10,000 steps or whatever, but they should be much more seamlessly integrated, and the measurements are not driving the intervention in a way. For example, the things that we are measuring is not guiding CBT or ICBT and some of it is not even well-aligned, right, so yes, there are measurements of sleep and energy which could be triggered, but what kind of intervention is well aligned with the measurements, and I don't think we've thought about that enough, we just kind of say we do CBT, but sometimes we should explore it more broadly. That's it.

Female Speaker:
Fantastic. Thank you, Tanzeen. And I think that one of the things I'm hearing in what you said is accounting for human complexity and comorbidity across psychiatric disorders when we're looking at, sort of focused on prediction in one area, whether that's taking that into account. So, finally, Dr. Moren from CMU.

Louis-Phillppe Moren:
Yeah, thank you very much. It is a great pleasure to be here. I'm a computer scientist by training and probably by heart [unintelligible] was a psychiatrist or psychologist. I just want to bring up the point that Brian made earlier. We talked a lot about sharing data. We didn't talk as much about sharing what we will call train model with these parameters, easy to replicate model from machine learning, and I think that's one of the issues that is being raised in machine learning probably 10 plus years ago and has been changing. I don't see it's changing as quickly in healthcare. People don't share not only the data that we some for the data, but also sharing the model.

My heart isn't multimodal also and multimodal is the science of heterogeneous data. This is including also temporal heterogeneity. So, when we think of heterogeneous data, we talk a lot about the mobile phone, but I also want to bring up there are other modalities as well. We have a lot of emphasis these days on telemedicine and their modality is retrieval communication, verbal, vocal, and visual, and I think we want to also think about these modalities.

And I want to bring, because we discussed like machine learning, machine learning and deep learning have great advantages. They can often learn from large amounts of data, deep neural architectures, but they have big challenges that they bring in compared to typical statistical approaches, interpretability is a big one, another one is fairness, and the third one is personalization. So, it's much harder in a deep neural architecture to do interpretability of the result, why is it working. Fairness is hard, like is it working because in my population there were more men than women, and suddenly, I have a bias, and that was also one of the points that were mentioned in the question.

And finally, personalization, we talked about using the same system in many different hospitals, but we need to be able to quickly and efficiently personalize it and customize it. And I will just end with the idea, which is not completely new but has not been emphasized yet so I want to bring it, the paradigm we talked about, human knowledge, we talked about machine learning, but we didn't talk as much about these hybrid human and machine learning where the learning happens together and that there are a mix and a collaboration. So, these are new and interesting paradigm and that’s a new oncology discussion, and that's new because I brought it, but I wanted to bring it into the discussion. Thank you.

Helen Egger:
Thank you so much, that's fantastic, and I would add to your comment around personalization, that it's not only the issue of personalization for different systems or health systems, it's also how you translate prediction done on the group level to prediction of individual trajectories and the individual pathways so it's reflected there. I know we have some questions. One of the questions I saw —

Sarah Lisanby:
I'm sorry, Helen, I just want to break in to do a time check. We're going to have about five minutes before we go on break.

Helen Egger:
Right. Right.

Sarah Lisanby:
And we have Susan Azrin from NIMH who's been monitoring our chatbox and [unintelligible] box and has cued up some questions to lead off the discussion.

Helen Egger:
Right. Great.

Sarah Lisanby:
Susan, you want to go ahead.

Susan Azrin:
Yes, thank you. Can you hear me?

Helen Egger:

Susan Azrin:
Great. Thanks. So, this question is from Holly Campbell Rosen to any of the panelists who'd like to jump in. Could one imagine a tool where providers and patients get guidance and then in nearly real time interact with a predictive system in the clinic creating a closed loop that incorporates adverse effects, treatment preferences, and the like in which may adjust the decisions?

Greg Simon:
Yeah, this is Greg Simon, what I'd say is, yes. I mean, you know, that's what we would hope to build in terms of guiding depression treatment and I think we will get there but as I said, there are some key things we need to develop in terms of the kind of better data we need. You know, as I'm fond of saying, we talk a lot about omics, genomics, metabolomics proteomics, but we need to think more about preference-omics and value-omics in terms of how we incorporate what people want and how they feel into decision making. I think that's a — the technical challenge there is not that difficult we just need to prioritize it.

Peter Zandi:
I would just add to that, this is Peter, that to get to that vision I think Roy's comments are pretty prescient here that will require equal parts science and equal parts engineering to develop that kind of solution to be able to develop those predictive models and to develop the engineering to deliver those in a way that's useful to both the clinician and the provider. So, I think Roy's comments in this regard are really on target.

Helen Egger:

Rebecca Rossom:
This is Rebecca. I'll add that one way we found around that is to develop web-based decision support tools that to the user look like their part of Epic and they're integrated with Epic. So, the things that the clinicians do in Epic inform the risk tools and that creates that loop. But because it's web-based, it's not a change [unintelligible] quickly as guidelines change or other information changes.

Roy Perlis:
We call it sidecar approach and that's been our strategy as well. I think that the difficulty is you got to remember most of psychiatry is primary care, right, and so while we as psychiatrists might sit around and look at the tool and think about this and a little bit of that, most of PCPs I talk to say just tell me what to prescribe and then make it easy, let me click a button, and that kind of integrates with Epic, I think is where the engineering piece comes in. We got to make it, so we present recommendations they press a button. It doesn't mean they shouldn't have the conversation, but in the real world it rarely happens so we have to acknowledge that.

Helen Egger:
Holly, do we have time for one more question, or should we —

Sarah Lisanby:
Yeah, Susan, I think we have time for one more question.

Susan Azrin:
Great. This is from Daniel Handworker [spelled phonetically]. It's for Greg and Rebecca, the NHRN. The question is, the NHRN data collection seems to focus on patients with risk of suicide and self-harm and this points to a general issue. Are there data collection biases in these systems for patient populations that don't engage with the healthcare system as regularly?

Greg Simon:
Yeah, this is Greg, I'd say, you know, the data that are theoretically available to us include the entire data exhaustive healthcare, from every healthcare encounter received, you know, probably 25 million people. We typically will create purpose-built datasets for a particular job. So, the job of suicide risk prediction we harvest data specific to suicide risk prediction for people who are at elevated risk, but the native data are comprehensive.

All that said, they reflect healthcare and as we know, especially regarding mental health problems, there are people who never seek care and their conditions may go completely unrecognized and those people may differentially come from traditionally underserved racial and ethnic minority groups for instance. So, yes, there are biases in the data available to us reflecting the biases in access to health care in this country.

Sarah Lisanby:
Thank you. Helen and Guillermo concluding remarks?

Helen Egger:
Guillermo, you want to go, speak first?

Guillermo Sapiro:
Let me just thank everybody and just maybe for the future discussion. I know there were some questions about interpretability that Greg had some great comments. I'm sure it will come back. I just want to put that sometimes it's not about interpretability, but it's about transparency, and that's something that we also need to put forward. And I want to thank the panel, I want to thank the opportunity that NIH gave us and wish everybody stay safe and continued learning this afternoon.

Helen Egger:
Yes, and I echo that and thank you so much, and I know already there have been things I've learned, and questions raised that I hadn't thought about. So, thank you to each of our presenters and to our discussions.

Sarah Lisanby:
Great. Well, thank you, Helen. Thank you, Guillermo. That concludes our first two morning sessions. We're now going to take a 15-minute break. We will resume right on time at 11:30 a.m. Eastern Standard Time, and please take this as an opportunity to stretch and get some food and coffee and fuel up because what the — we have even more exciting discussions to come. And those of you who did not have your questions answered, we will circle back to those during our one-hour general discussion session between 1:30 and 2:00 p.m. to 3:00 p.m. Eastern time. So, we're going on break now and we'll resume at 11:30 a.m. Eastern time. Thank you so much.
In about three minutes.

We're going to be resuming. I just wanted to do an audio check for our next two moderators, Jen Goldsack and Jeffrey Girard [unintelligible].

Jen Goldsack:
Hi, Holly, this is Jen.

Sarah Lisanby:
Now, do we have Jeffrey Girard.

Jeffrey Girard:
Yeah. How does this sound?

Sarah Lisanby:
Perfect. So, welcome back. Again, I'm Sarah Hollingsworth Lisanby from the National Institute of Mental Health and welcome back to the second half of our virtual workshop on transforming the practice of mental health care. I want to thank the participants from the morning sessions who really helped set the stage for where we would like to go in terms of our future vision for providing supplies and effective care for every person wherever they're receiving treatment and laying out the lessons learned from [unintelligible] currently underway to mobilize a large data set multimodal data to inform care.

Now session three is where the rubber hits the road. We're going to devote this session to learning — to discussing how do we get from where we are now to that desired future state. I'd like to thank our moderators, Jen Goldsack, who is the — leads the strategy portfolio and operations at Health Mode and serves as the interim executive director at Digital Medicine Society [unintelligible], who is joined as co-moderator by Jeffrey Girard, who completed his postdoctoral research associate at Carnegie Mellon University and is now an assistant professor and Wright [spelled phonetically] faculty scholar at the University of Kansas. I'd like to without further ado hand it over to Jen Goldsack to kick us off.

Jen Goldsack:
Thank you, Sarah, and Jeff, I think you're going to do the run-in for us today, right?

Jeffrey Girard:
Yeah, so we'll talk a little bit here. So, in this third session, we will be discussing the intimidating but also the thrilling task of trying to turn these technological promises and possibilities into realities that are both scientifically grounded and clinically feasible. In walking this path forward, it is going to present many challenges that we have to overcome. Some are unique to specific subareas like genetics, or you know, different areas that each of us works in, and then the other ones are probably going to be common across many different areas, and so we're hoping to explore all of that. And I'm excited to have a really fantastic panel here today who really represents a wide swath of areas from academia, medicine, industry, and government to discuss their thoughts and especially their kind of in-person experiences in, you know, kind of trying to blaze these trails forward.

So, each speaker is going to make a brief presentation, we're hopping around three minutes, maybe a couple of minutes longer, but we'll try to keep you short, and then we'll have a broader discussion where again we're looking for those unique, as well as more common themes that emerge from this. So, with that in mind, we'll start things off with Glen Coppersmith, who is the founder and CEO of Qntfy, and we'll be speaking about quantifying mental health signals from digital life.

Susan Lisanby:
And Glen, you may be muted. Okay, I believe that Glen may have lost his audio connection. Glen, can you hear us? And I might make a suggestion —

Jen Goldsack:
In that case, why don't we skip ahead? Brian, why don't you dive in with your presentation and then as Glen jumps back in, we will revisit his presentation. So, thank you for advancing the slides, guys. I really appreciate it. So, Brian, if you want to briefly introduce yourself and then dive straight into your presentation on intermediate biomarkers.

Brian Pepin:
Sure, can you hear me?

Jen Goldsack:
Perfectly, thanks, Brian.

Brian Pepin:
Great. Yeah, so my name is Brian Pepin. I'm an entrepreneur working in the sort of software and data analytics around neuroscience therapeutics, specifically neuromodulation, TMS, DBS, SCS type therapeutics.

I'm going to talk about — I'm going to this quick example to just demonstrate a concept that I think is important for the panel, which is that if you're talking about developing therapeutics, maybe especially neuromodulation pharma therapeutics for psychiatric conditions or other kinds of brain disorders from depression to autism and sort of everything in between, behavioral endpoints are really tough to use because they're noisy, they vary from rater to rater, there's all kinds of other factors that go into play and everybody is sort of familiar that why those things are difficult.

So, what we're seeing is a movement towards using a variety of metrics ranging from sort of imaging, EEG, FMRI type metrics, all the way down to sort of digital health activity type metrics to augment, I think initially, and then ultimately replace these behavioral biomarkers as endpoints or replace behavioral endpoints with these new kinds of more quantitative biomarker defined endpoints.

So, this example is for Duchenne Muscular Dystrophy where the neurodegenerative disorder in children and the conventional endpoint for this is called a six-minute walking test, which has a lot of the behavioral elements to it. The patient basically walks back and forth over this six-meter course, over six minutes, and then you record how many times they sort of walk back and forth. So, there is — even though there's an element of quantification here a lot of behavioral elements they get they sort of bleed in.

So, it's stereotyped — heavily stereotyped, patient motivation is known to play a very major factor, there's all the normal stuff around the patient has to, you know, come into the clinic and patients have movement disorders that can be difficult, and then as people mentioned in the last session, you would get this snapshot effect for a condition which realistically is fluctuating over time.

If you can advance to the next slide. So, what some of our colleagues at Genentech Roche have done in this case is taken that six-minute walking test endpoint and replaced it with what they call a 95 percent stride velocity endpoint. So, they use wearable watch-like device that's actually — the children were on their ankles and they [unintelligible] sort of the normal course of daily life and they're sort of basically they would run around as they would normally essentially and we're able to text when they're walking and it measures this metric, which is called 95 percent stride velocity, so it's not the peak stride velocity, but it's just often.

There are various reasons why they're doing that, but the end result is what you have is continuous, so you're measuring a bigger picture of the patient's whole experience where you're quantifying it more accurately without these confounding things of like motivation and having to travel into the clinic, and what they found is that the effect size for these two tests is somewhat similar, 7 percent versus 8.5 percent, but the standard deviation for the stride velocity test is far lower. So, it's 7.9 percent versus 20 percent, and what that translates to, in terms of therapeutic development, is an N of 14 in a study time of six months for the stride velocity endpoint versus an end of 112 and something like 12 months for the six-minute walking test endpoint.

So, they've actually gone and started the approval process for this new endpoint in Europe and they're awaiting some final decisions and then they're also working through the FDA to get this. So, they've actually taken even the next step of saying, “Okay, we have this sort of [unintelligible] or intermediate quantitative biomarker; we're actually going to go and get this. We feel like we have enough evidence to go ahead and approve this as the primary endpoint for a study.” And then once it's approved for the primary endpoint, you know, in this case this is a somewhat rare disease so it makes it much easier for them to recruit for clinical trials, because it can be a smaller trial and it can happen faster and so it can increase the speed at which they can iterate on drug development and make it more cost-efficient. So, it's — that's real money there in terms of those clinical trial numbers.

So, that’s kind of the example I wanted to go over and then there are many, many different examples of this for how these kinds of markers can make a difference in autism, drug development or depression drug development but this one was just a really nice, kind of quantified example, so I wanted to share it.

Jen Goldsack:
Fantastic. Thank you so much, Brian, and Glen do we have you back on the line?

Glen Coppersmith:
Maybe? Is everybody hearing me?

Jan Goldsack:
Yes, I heard that.

Glen Coppersmith:
All right. I'm sorry about that.

Jan Goldsack:
No problem. I'll let Jeff introduce you as these slides come up and just to note for everyone on the line, we'll run through all of our presentations, will then have a short discussion, and then we'll take questions. So, if you have questions for our presenters as they go, please feel free to enter those into the chatbox, and with that, Jeff, I'll give it back to you, and thanks again, Brian.

Jeffrey Girard:
Yeah, thank you for being flexible here. So, Glen, yeah, why don't you introduce yourself briefly, and then go through your presentation on quantifying mental health signals from digital life?

Glen Coppersmith:
Thanks for that and I apologize for the audio issues, it's always the tech guy that has the issues, right. So, I'm Glen Coppersmith. I'm a founder and CEO of Qntfy. We're a tiny company that couldn't afford the vowels when we got started, but we're — I've lived in this world in between computer science and psych my whole career and so this is sort of the outpouring of that.

Next slide, please. So, what I want to briefly touch on here is what we're facing is a Wicked Problem in the Rittel and Weber sense, right? This is not a binary decision as to what medication we should give someone, this is not a binary decision about what treatment we should give them, but it's actually more like trying to drive an autonomous car, we're not going to pick a point on the map and take a picture of the trajectory from here to there and use that as the sole source of navigation.

This is actually much more like the continual feedback loops that we want. So, as things are changing in the world, not that there have been many major changes in the world recently where things are pretty steady as far as I understand it, but as things are changing in everyday life and as people's personalization, their desire, what they need out of treatment is going to change.

And so, one of the fundamental flaws in all of the applications of AI machine learning in this space is that we actually need to think about this in terms of the continual measures and so how do we actually get from here to there instead of the binary distinction. And so constant — this is — the way you're solving problems are small compounding iterative improvements in measuring constantly, and that constant measurement is the piece that's missing.

Next slide, please. So, there is some work here. We've done some work in the past looking at those lines about estimating suicide risk from data from outside the healthcare system and so it's wonderful to see a couple of people had a version of this plot were looking at the healthcare system interactions and other non-healthcare system data are present there, right, so if all we're talking about is using EHR data we're in some ways limited to looking at what happens within the healthcare interaction, and so this is one person over three years, and you'll see up top in red their health care system interactions, in blue, they are post on social media, and in gray their wearables or Fitbits.

So, what this is — what we've been looking at, we do see there's a whole — there's a bunch of literature, not just the stuff that Qntfy has published, I'm looking at these things and there are great measures here and what we really need to be able to do is to get the humans to trust these measures. And so this is the largest challenge that I've run into over the last seven years that I've been doing this, is that convincing the clinicians that this is a reasonable [unintelligible] and there is useful information here, you know, to fill in that whitespace has been the sort of fundamental challenge. So, from my perspective, the tech is solved or solvable, the really hard part is the interaction with the humans. So, with that, I'll step back. Thank you very much.

Jeffrey Girard:
Oh, that's great. Yeah, thank you for staying on time and for your thoughts on that. I really like that metaphor of the autonomous vehicle and this being a manual process. I think next, we will have John Torous, who's the director of the digital psychiatry division of Beth Israel Deaconess Medical Center, and he'll be talking about mental health app uptake and clinical integration.

John Torous:
Thank you, guys. We'll go — I'll jump right and maybe ask to go to the next slide. I think that there's a lot of interest from our patients and clinicians in using mental health apps to kind of augment and extend care. What's tricky though — and I think the metrics we have today about uptake and use are not accurate.

On the left, you can see a paper from 2016 that showed up star ratings and the number of downloads really means nothing about health app and quality. On the right, you see a very interesting paper in 2019 looking at people in the public who download a mental health app. How many people are actually still using it two weeks later? The answer is about four percent. The one app group that abuts the trend is pure support in gray, those apps have about 18 percent engagement, so again, I think we have to be honest about who are we reaching. It's not just that it's on the App Store, that's good, it's kind of what is happening and how do we pick it?

We'll go to the next slide. So, I think, as we kind of — as Glen just told us, there really are some issues around trusting these things. This is, again, an article screenshot I took from March 9, 2020, that says is mental health app threatened to sue security researchers over a bug report, so I think we do kind of still have some security issues in this that are kind of holding back trust.

On the right is a paper we published in NPJ Digital Medicine where we actually read app stores and looked at what is the evidence that these apps are claiming or descriptions? It was about 64 percent were kind — 44 percent were using scientific language about their app. We track that evidence down, it really was about 1.4 percent. So, again, there are a lot of reasons for people to be a little bit skeptical.

Next slide. So, we’ve done some work with the American Psychiatric Association, not recommending any app, not endorsing the app, but kind of giving people evidence-based principles and guidelines, a framework to ask questions about to think about making a smart choice in these apps and you can see it really starts with privacy and security and then clinical foundation, which is evident, and then thinking about engagement and then kind of thinking about the therapeutic goal.

Recently, New York City — actually, before its pandemic — could have used this framework to make its own app library. So, we're not saying which apps you should pick, saying look, just ask the right questions, make smart decisions, all the things we've heard this morning let's put this evidence to use and framework so that people can use to kind of use the right app and have good experiences.

We'll go to the next slide. So, our team actually does a lot of work on digital phenotyping. That's actually our main thing about predicting relapse in psychosis with smartphone sensors and that research is very exciting. In part, we can actually use elements of that today and we'll call it a digital clinic. So, you can see the kind of, on that 1, 2, 3, 4, 5, those are different digital data streams that we can collect in research, but also from someone's step count whose has passive data showing what call and text logs could look like. We don't know what people are saying or the content, but we can see kind of what days people were more socially active on calls and text. We can have people tagged their environment number three, we can have people take real-time survey number four, we can begin to do kind of simple inference in predicting their anxiety today, and tell us about, say, your mood tomorrow.

What we do actually to kind of take this outside of research, which again, we're doing to put it into clinical use is, we work with a patient in our clinic and we say, “You know, what is the clinical question? What data would be useful for us to learn about?” And you can kind of see on that flat icon diagram, we can say, “Hey, are we trying to learn about anxiety and sleep; let's track those for the next visit and add a survey. Are we trying to learn about mood and mobility patterns? Let's track that.” So, for each patient, we can kind of customize what we can do and what we can collect.

We'll go to the next slide. This slide isn't meant to overwhelm people, and the app platform we use that's open source, we have no commercial venture. We offer people ways to learn and talk about managed prevent, and it's overwhelming. There's a lot of different options you can use and for each patient we don't want to use all the things, we want to customize what people can do, and I think that's where the clinical integration really comes in. Not everyone needs — every sensor needs kind of everything we can do. We can pick the right ones for the right people.

So, we'll go to the next slide. In part again, this is a kind of schematic of how we kind of run the digital clinic. On the left, we still put the patient and clinician together, supporting a therapeutic bond is the most important. And the technology part kind of as an addition, we've actually created a new team member that we call a digital navigator, just like in radiology, you have a radiology tech and in pathology, you have a pathology tech. It's probably time for mental health to have our version of kind of a technologist and we call it digital navigator really because there's a lot of patient interactions, there's a lot of supporting therapeutic alliance, and then that schematic on the right, you can kind of see different things that we do in our clinic that the digital navigator supports in this role.

And I'll go one last slide. We also think it's important to be teaching our patients with serious mental illness how to use technology. Again, we do a lot of work in early psychosis, in schizophrenia, and bipolar disorder and patients may increasingly have phones. They've been given them; their social worker gave them a phone. They don't really think of their phone as a tool that can promote their recovery. And so, hosting groups that kind of teach people how to use their smartphones as meaningful tools and help and give them the skills and competencies really can be a very powerful intervention just on its own. So, we have this manual app called Doors digital outcomes and recovery services that's an open source, it's where you can look at it, we're always looking for people kind of helping us do that as well. So, with that, I think I am out of slides. Thank you.

Jen Goldsack:
John that was terrific. Thank you so much. So, next, we'll hear from Akane Sano. She's going to talk to us about sort of the barriers and challenges as we try and move towards data-driven mental health care. Akane, if you could introduce yourself to our guests and then dive right in. Thanks so much.

Akane Sano:
Hello, everyone. My name is Akane Sano. I am the chief of professor at Rice University. I am an engineer and computer scientist working on developing machine learning — personalized machine learning model, especially for mental health in sleep using multimodal physiological and behavioral data.

So, yeah, since this morning, we have a lot of discussion about data, and I just want to briefly talk about the challenges as engineering in computer scientist to deal with this kind of dataset for mental health research. And we have various kinds of the different datasets from different sources in critical data, some digital data, smartphone, computer usage, sensors, internet services. We also have self-reported.

We have various kinds of challenges, for example, is it better to collect more data? What is the burden of users and the investigators who are [unintelligible] to collect data? And we have a lot of noise, especially the data we are working on is coming from their daily life setting, so we have some missing data and noises, and how can we deal with it, and as we already discussed, the user and patient adherence is another issue.

Also, physiologically behaviorally, the patient-user status has been changing over time. We have to think about how we can deal with time-varying status. We also have a lot of individual differences so some data — so the labels, clinical labels or self-reported data include very skewed information. That means the distribution might not be, you know [unintelligible]. We have another skewed data around high scores, low scores. And also, if you think about, for example, predicting relapses, suicidal events, those events are very rare and compared to the non-rare event, it's very hard to detect and catch those rare events.

Next, please. So, if we think about machine learning models and data science approaches for mental healthcare, there is a lot of ongoing research. For example, we want to classify, or we want to diagnose mental health disorders, we also want to detect or predict patient symptoms, risks or even not for the patient. But also, we are interested in how we can [unintelligible] how we can detect people who might have a high risk in mental health disorders.

Not only detection, prediction, diagnosis, classification, but also, we are interested in selecting treatment intervention for the right users at the right timing. And a lot of ongoing research has been trying to develop machine learning models, and also, recently, one of the things we have discussed in a previous session is personalization. Personalization is the one of the key elements, especially because everyone is very different, and our situation context trait physiology is changing over time.

Then not only personalization, but also, we have to think about how to think about the subtypes of groups of people because everyone is different. But it's difficult to get a lot of data from each individual. So, if we think about the subtype people into subgroups, then we might be able to get a better model. Another challenge is dynamics. As I already mentioned, people's physiology, behavior, or preference conditions have been changing so our machine learning model has to be also dynamic; however, a lot of models we have developed so far have been still very static.

Next, please. So, if we think about what the ideal data we can collect and we can use for our research or treatment, the data has to be of course, like, long-term, and we really want to have data from diverse populations of patients and users in terms of demographics and geographic regions. Again, some of the clinical events can be very rare, so we want to have more data of those clinical real events. And since many of us are interested in integrating that multimodal data source, so this data format has to be consistent and standardized.

So, these data will definitely help understanding individual in subtype differences. We also need to think about how to build some environment of the building and testing dynamic and personalized machine learning models to capture time-varying user symptoms and physiology and behavior.

Next, please. Then another thing we have to think about is, so a lot of us, like, in data, [unintelligible] researchers have been developing a lot of models, but how much precision and reliability and interpretability that we really need in building clinically useful and trustful tools. So, this is not about accuracy, precision, but also what kind of outcomes should we provide through this kind of model? What kind of tools and information are really useful for clinicians, patients, and other stakeholders?

Deep learning has been very successful even in the healthcare setting; however, a lot of people know that we sacrifice some interpretability in [unintelligible], so how can we take the interpretability versus precision. And also, another thing I want to emphasize here is we really need to build some research environment to integrate comparative data set from multiple studies and multiple sites and also some research environment to accelerate developing and testing the effectiveness and safety of machine learning models in our findings.

Some of the healthcare research, for example, ICU data, like mimic database in the physio net, the physiological dataset has been public, and this has been successful to increase the number of research activities. So, in like mental health research, I am really hoping that other kinds of things will be happening, and mental health will be affiliating these kinds of activities. Thank you.

Jeffrey Girard:
Okay, so now, I think we will move on to Matt Switzler and James Hickman, who were both coming from Epic. And we're excited to be able to kind of hear from them directly about, you know, working on the electronic medical record.

Matt Switzler:
All right. Can you guys hear me?

Female Speaker:

Matt Switzler:
All right. Hi, everybody. My name's Matt Switzler. I'm a software developer on the behavioral health team here at Epic. At Epic, we provide electronic medical records, as well as lots of other systems that health systems can use to help their operations. Lots of the other presenters have talked about the types of data that you can get kind of in the traditional model of clinical decision support that is typically implemented by a human that needs to have a robust research behind it to have the appropriate interventions.

And we've seen lots of good outcomes for these systems in data-rich environments, but one of the fundamental struggles that we've been hitting in the mental health space is that unlike when patients have physical medicine problems, patients deteriorate, for physical medicine conditions, those — they tend to engage more with the healthcare system when you can collect more data points about them. Whereas if somebody has a mental health problem, they tend to engage less with the healthcare system as well as many other parts of their life, which gives us less data to work with. So, even if you are sending out things like questionnaires and that sort of thing, there's no promise that you can get that back.

That being said, even with some of these limitations when we've been able to see some pretty good outcomes in areas like where do you see no show rates, determining readmission risk, as well as the early detection of physical medicine diseases. For example, we've done a lot of cool work around sepsis, so the infrastructure is in place. And it's making a different kind of today, and we have active work ongoing to help augment that with AI and machine learning and we're glad that we can be at a forum like this to prove this in the behavioral space. I'm going to pass it over to James, and we will want to advance next slide, and he can talk about what we're doing to augment these systems with AI in Epic.

James Hickman:
Thanks, Matt. So, as Matt mentioned, our focus of experience at Epic is really the application of machine learning at the point of care and that includes the acute care setting, of course, the ambulatory clinic and then increasingly, outside of the health system, [inaudible] for population health.

The early [unintelligible] industry being machine learning within healthcare has focused on helping and improving workflows that already took advantage of scoring tools with more accurate machine learning predictions. But I think in the last couple of years we do see a little bit of a feedback loop emerging as health systems are starting to look at leveraging machine learning for their informatics to enable new workflows. And some recent examples of that are, you know centralized monitoring in the hospital setting, and also a large number of specialty-specific predictions that are putting more power in the hands of a primary care provider, others frontline providers prior to the console and to the more limited specialty resources, be that nephrology or perhaps behavioral health.

There are some key concepts that we found are important to ensure the success of these efforts. A lot has echoed already this morning, but first there are providing tools that are transparent and, of course, that's important from a quality and an ethical standpoint, but it's also a key to success for increasing the actionability if you will of these tools. So, in our experience, this has been a little bit less of a concern over the kind of arms race of the type of algorithm that we're choosing and it's much more round user-centered design in exactly how we present that prediction to a particular user. The second is putting the prediction in clinical workflow. So, catching that patient when we have them with us for a visit, perhaps also taking advantage of brand-new clinical documentation and rapidly, you know, revising a patient's risk for a particular event in real-time.

And then last, I certainly agree that we must put a premium on the experience of providers that we're asking to use these tools. So, there's really an emerging science in designing both UI in models that aren't solely optimizing towards accuracy, but really the impact on clinical workflow. And to that end, certainly, I'll support some of the statements made earlier about the difference between something that varies [spelled phonetically] under the curve as a method of evaluating models, and particularly emphasizing some of the threshold metrics like positive predictive value or negative predictive value.

So, our development at Epic and the opportunities that we have to work with health systems that are clients of Epic has really focused on making the deployment as technically easy as possible, and while maintaining intellectual property rights. We certainly welcome conversation today or afterward about the role that Matt and I and others at Epic as developers can play in motivating and supporting the spread and sharing of bottles.

We understand that there are data challenges, we understand that there are also intellectual property questions and sort of validation on different populations and things like that, happy to participate in all of those. So, I really appreciate the opportunity to listen in and participate with so many leaders in this space. Thank you.

Jen Goldsack:
Matt and James, thank you so much for joining us today. And let's move on to our final presentation of this session before we dive into our discussion. We welcome Donelle Johnson, to talk a little bit about the importance of the workforce, keeping pace with this era of technology. Donelle, if you'd like to introduce yourself and then dive in, that would be great. Thanks so much.

Donelle Johnson:
Sure. Thank you. I'm Commander Donelle Johnson. I work in the National Substance Health — Mental Health and Substance Use Policy Laboratory at SAMHSA. And so, I'm just going to talk to you about some of the challenges and opportunities with workforce as far as transforming the practice of mental health care.

You can go to the next slide, please. Okay. So, given SAMHSA charged to reduce the impact of mental illness and substance misuse in America's communities ensuring that individuals with mental health have access to evidence-based, high-quality care is a critical focus for us. A major factor in achieving this goal is addressing the training and education needs of our practitioners. To facilitate the achievement of this goal, data on the current numbers and trends within practitioners is imperative.

So, we have the mental and substance use disorder practitioner data program, which provides comprehensive data and analysis on individuals who comprise the prevention and treatment field to address mental and substance use disorders. So, the goal of the program is to provide valid data on existing practitioners and usable information to SAMHSA on which to make policy and planning decisions.

For instance, knowing how many and the variation of practitioners can help us determine the needs and areas to focus on. For instance, we know traditionally minorities are underrepresented in this in behavioral practitioners, so we have the Minority Fellowship Program, which supports the development of behavioral health practitioners, and specifically it aims to increase the presence, knowledge, and skilled practitioners available to serve racial and ethnic minority populations.

By increasing the number of culturally competent professionals in the workforce, the program seeks to reduce health disparities and improve behavioral healthcare outcomes for underserved minority communities. The program also seeks to encourage racial and ethnic minorities to join the behavioral health workforce. The relative scarcity of professionals who are from culturally and linguistically diverse backgrounds constitutes a workforce issue that contributes to the current disparities in quality of care and access to behavioral health treatment.

We also have the technology transfer centers, and so, these centers helped to develop and strengthen the specialized behavioral healthcare and primary health care workforce that provides prevention, treatment, and recovery support services for substance use and mental health disorders. The mission is to help people and organizations to incorporate effective evidence-based practices into substance use disorder in mental health prevention, treatment and recovery services. Together, the network serves all 50 states, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, Guam, American Samoa, Palau, Marshall Islands, Micronesia, and the Mariner islands.

Now through the centers, we provide technical assistance to all providers, not just SAMHSA grantees. So, within the centers and the training programs that I mentioned, there are opportunities to educate and train providers on using technology, which can address some of the barriers. For instance, it can help to increase providers knowledge and comfort with using the technology as well as them being able to gain an understanding of the patient's knowledge and comfort in using that technology, also helped them to acquire the knowledge about privacy and reimbursement policies around the use of technology, especially with things like telemedicine and all the rules around how you can get reimbursed for that.

While there may be barriers such as cost and infrastructure to using technology both on the provider end as well as the patient end, training and educating the workforce to effectively use technology and practice is an opportunity for improving access to care. So, that’s about it, but I just think we can't forget the importance of helping the workforce to get on board with this so that they can move it forward. So, thank you.

Jen Goldsack:
Terrific, Donnell, thank you so much. And just to orient folks on the line for how we're going to handle the rest of the session. Jeff and I will lead some quickfire questions for our panelists and then we'll throw it open for those of you on the line to raise questions directly to our experts. And I'll kick us off, Jeff, if I can, and dive in. Glen, I have a question directly for you.

During the prep work, we had some discussion about sort of what the limiting factors are for introducing these sorts of digital measures into this sort of practice of mental health care. You and I were on the same page that we don't believe sort of technology is the limiting factor. I wonder if you could talk to us about what you believe some of the other considerations are beyond the technology itself. What's holding up the implementation of these digital tools?

Glen Coopersmith:
Yeah, thank you for that. I agree. I think technology is an assault on the grand scheme of things here and the larger barriers are cultural ones and their ethical ones. And so, it's not — I may not say what you're going to expect me to say, is that the patients want more analysis in different things. They see what they're getting for targeted advertisement and that that works stupidly well and then they turn around and none of that data is being or little of that data is being used to help them. You know, it captures a lot of the same data, but it's not being used to help them understand their well-being in helping to make them better.

So, this has — I think this all comes down to a single point that we can look at and it's a very interesting ethical discussion we've had with a number of colleagues in that this is a very interesting spot we're putting privacy against the duty of care and so if we have technology that's capable, for example, of predicting suicide risk from all the public data that we can find about a person or even more interestingly, people that have opted into it, if we could predict the suicide risk it's going to outstrip the capabilities within the healthcare system.

This is now putting the privacy of the patient who — they've given it away. They're willing for us to analyze them. They're willing to be analyzed, putting that against the duty of care for a system that is, there's a real cost for not engaging with these sorts of things, and so this is the — so often it's a one-sided thing or the privacy of the patient is paramount and I agree, but there's also this other side of that coin where there is a cost in terms of lives if we're not going to use the technology available to us.

And so that all comes down to this, that is the point of this like large ethical, everyone screams that there are ethical issues here and there might be, but at the same time, like, we have to look at both sides of this ethical issue, not just that the technology is somehow scary.

Jen Goldsack:
Glen, that was terrific, and a really sort of provocative teaser, I think, for our next session on ethics. So, strong work on the segue there. Thank you, Glen. Akane, I wonder if we could pose the next question to you. You spoke a lot of having the right data in order to do this well. What else do we need to build a sort of clinically useful and trustworthy tools in mental health?

Akane Sano:
Hi, I think in addition to the data we need different, you know, some system — some ideal online system for researchers to test with built-in test models like in real-time. And also, the other important thing is, you know, we also need to communicate closely with clinicians and users in what they are looking for and what kind of user experience they have. So, I'm an engineering computer scientist, but I also closely work with clinicians. We also need to integrate this, like working with various stakeholders in getting input into our system and models.

Jen Goldsack:
Fantastic. Thank you, Akane. Yeah, you got a couple of questions for our colleagues [unintelligible].

Jeffrey Girard:
I do, yeah. So, I'll — in the next one at Brian, we had talked about quite a bit in the whole workshop today about this idea that, you know, these new technological measures could help us actually inform more tailored therapeutic regimens. So, I'm hoping you can speak a little bit to, how does the rubber actually meet the road on that? How do we integrate these measures into, you know, tailoring the therapies that we actually provide?

Brian Shiner:
Thanks, Jeff. So, kind of two — I'll go two elements here. So, one is you have to have the right end goal in mind, and by that, I mean, ultimately, you're trying to get an FDA approved therapy with this sort of intermediate biomarker, maybe a companion [spelled phonetically] diagnostic, or you're getting an FDA approved biomarker linking some physiological output to some integral trade or some clinical diagnosis. So — [sneezes] excuse me.

So, that implies a lot of sharing of your data with the FDA, building to a big consensus, these things generally aren't patentable. The link between a physical signal and a sort of physical outcome aren't typically patentable, so you have to kind of have that framework in mind. You know, if you're a company, this biomarker is not going to be your competitive advantage, right? You can't kind of have it both ways and have a biomarker to your competitive advantage and also something that there is broad scientific consensus in the community such that it could get FDA approval and actually be used as an endpoint.

Those are sort of mutually not compatible. You kind of — similar if you're an academic group. You can't, I think, have a very big impact if you're just sort of — you know, creating sort of datasets, which you don't consent in a way that they can ever leave your — you know, the identified way — ever even leave your university servers, and then, you know, your group can continue to publish off those papers, but it's very hard to replicate. Very hard to integrate that with other work that people are doing, and you never build this broad consensus that this is a real biomarker. So, I think starting from that philosophical place, which is, I think, there's still a lot of people that are all across the map there is a good beginning.

And then practically, to get these things actually integrated into studies, basically just realize they don't come for free. Like, the infrastructure to do this, the patient software interfaces, the platforms, the actual hardware devices, none of that is free. And so, if you're biotech or a pharma company or MedTech company and people are doing this, you just have to budget for that and think, you know, "Okay, if I have this and I can run a clinical study in six months instead of 12 months, and with 14 patients instead of 50 patients, how much is that saving me? How much should I — what's the uncertainty there? How much can I budget for this?" [inaudible] and integrating that and, you know, finding the right partners. And there's — for every indication across psychiatric disorders and everything else, there are companies that partner with pharma and biotech to do these sorts of things.

On the academic side, I think there's some stuff that NIH can do in terms of properly incentivizing people to do things like have kind of open source or access layers to at least identify data so that datasets from different groups are a little more inner operable. Kind of proactively, you know — when they're thinking about funding studies, making sure that there's plans for, you know, this kind of layer of quantitative biomarker stuff to be done with [inaudible] that are using common platforms and that the data infrastructure is there so that data can kind of live beyond the study and not just be sort of aggregated [inaudible] not just kind of dumped into a bunch of file folders and, you know, posted in the NDA. So, I think that's —

Jeffrey Girard:
That's great, Brian. Thank you. Yeah, so really trying to find ways to align, kind of, the incentive structures that exist for both companies and researchers with what we need to move the field forward. I think that's a really critical point I'm glad you raised. For the sake of time, sorry to cut you off. I think I could move on to the next question for John. It was great to have somebody who is really on the front lines trying to do this stuff as a clinician, and so I was hoping John could speak a little bit to, what is the actual experience like, both for providers but also critically for patients, to actually try to use these types of digital tools? You know, what are the good and troublesome aspects of that, in your perspective?

John Torous:
I think that's a good phrasing, there's good aspects and there's troubling. I think that sometimes there's frustration because they'll be very interesting scientific papers, reports done about using these technologies, but when you kind of read the research you go, "Well, that was done in a way that really doesn't match how we're going to deploy it in clinical settings." Or again, if you pay everyone $100, they get an extra therapist, they get a yacht, and a jet, and a mental health app and they feel better, that doesn't kind of always answer the question. So, kind of having studies that may be rigorous, but again, because they're rigorous, we know as you kind of add active controls, these app interventions, that they're not going to look as strong.

But I think having evidence that's kind of realistic and applicable and generalizable is so important because people do want to pick this up. Clinicians want to offer the best services for tools, patients want to get the best outcomes, so I think we're all on the same page. But I think sometimes there's a little bit of — the hopes kind of get ahead of the evidence. That's I think also designing these things about how to fit into a clinical workflow.

In part, we did kind of design our digital clinic [unintelligible] the workflow to kind of make sure we could use apps and technology, but I think it's something we don't love to think about, kind of messy implementation details. And I think even very simple algorithms and sensors can go a long way when they're kind of thought about that implementation and those results can actually be generalizable, that our clinician colleagues — again, myself included when we're doing this — can learn from.

Jeffrey Girard:
That's great, thank you. Jen, do we have one more kind of specific question before we go to the broad ones?

Jennifer Goldsack:
Yes. So, for Donelle — and John, thank you. That was really terrific. I liked how you spoke about the evidence needing to sort of keep pace with expectations. Donelle, your workforce presentation was really excellent, and I wonder if you could go into a little bit more detail about how a more representative workforce can help better integrate sort of behavioral health into broader public health? Sort of traditional clinical care and research. And also, how can we think about the role of technology in facilitating that process?

Donelle Johnson:
Okay, sure, thanks. So, a more representative workforce can be used to inform public health, traditional healthcare, and research. These providers are able to bring real world experiences into all those fields, and a representative workforce is not just necessarily limited to demographics, but also specializations in the different workforces. So, for example, I forgot which provider — I mean, which presenter it was earlier that mentioned the need for more child mental health specialties.

So, again, they can help to inform that and address these areas that need more specialists and areas like that. They can definitely inform the areas that you mentioned. As far as technology goes, technology can absolutely help to better connect individuals and providers, keeping in mind we do need to address some of the barriers. But with the advance of the mobile technologies, healthcare services can now be provided in new ways. Look at it now like, you know, with the COVID-19 going on, we're using a lot of technology. Providers are using it. Some of them may not have been informed or comfortable using it but being forced to use it now.

So, I think with the representative workforce, if they're knowledgeable about technology, they can gauge their patients' comfort and they can know what to utilize. Maybe everyone, especially the older population, may not use smart phones, but it could just be a simple telephone call that the providers would be able to use. So, they need to be aware and trained on the technology.

Jeffrey Girard:

John Torous:
And I'll just, quickly, I think pure specialists, it's a really great opportunity to kind of have them help in this kind of technology implantation role. We have a new need where we have pure specialists who really can do amazing work in this space.

Jeffrey Girard:
Yeah, that's a very interesting, yeah, resource to kind of broaden that workforce even further. I'd forgotten, we have one more question we haven't talked — a specific question for Matt and James. You know, we had basically talked about, both in this session and in previous ones, about, you know, sort of a need for some standardization in terms of data and just the myriad challenges that come from working with data on the large scale, especially when it's unstructured. And so, something like Epic and, you know, the electronic health records systems possibly presents a nice streamlined way of obtaining that type of standardization. So, I'm hoping you can kind of talk a bit about what you guys at Epic are doing or what you personally see as being, you know, important steps on a path forward to that standardization.

Matt Switzler:
Yeah, I think there's a couple things, a couple different ways we approach this at Epic. I think probably the broad one is we do participate in groups like care quality, HL7, we follow the fire standard, and we're often commenting on any ONC proposed rules around inner operability so that even, you know, you might be an inside Epic system, but if you travel to a certain hospital we want to make sure that we can exchange data with those groups.

And we don't focus solely on medical information, we continue to push them on behavioral data, psychological data, as well as social factors to make sure that they're considering that. From an implementation perspective, we in general push all of our implementations to track data discretely. Kind of a rule of thumb we use is, if you're not going to get the data backup, don't bother putting it in. So, we try to think about some of those reporting components at the very onset before people are even in the system. Those are kind of the big things I'm thinking about. James, do you have anything to add to that?

James Hickman:
Yeah, a couple things. Kind of thinking a little bit about comments earlier from Greg and Roy, we have an opportunity to work with a lot of health systems that are maybe deploying their own models or, perhaps, at times taking advantage of a model trained elsewhere. And our approach has been to guide those organizations to document things as uniformly as possible. Overarching goals, absolutely, to reduce the technical overhead of sharing and spreading models. We would absolutely love to see further uptake of shared ontologies, like going for labs for example, but in the interim, working pretty closely with health systems to make sure that, as they're managing and transforming data, be it in a data warehouse or some other system, that they are taking approaches that reduce the overhead of sharing the model. Our kind of strategy has been, overall, to make that validation and understanding of how a model travels as easy as possible, because there's plenty of complexity on the operational side of evaluating how that model fits in to clinical workflows. Thanks.

Jennifer Goldsack:
Matt, James, that's terrific. We are very spoiled, I think. Our hosts have been curating questions from the audience. Let's break now and see what questions folks on the line might have for this sessions' experts.

Sarah Lisanby:
So, we have Jenni Pacheco from NIMH. Jenni, are you — do you have some questions for the group?

Jenni Pacheco:
Hi, yeah. I have one that's more of a comment, but for anyone else that's on the line, if you have some questions you can use the chat box or the Q and A section to post any questions. The comment I have is just following up on [unintelligible] point about the need for resources to train and test models. They are saying that there's an interesting example outside of the clinical domain for training deep R.L. models in the gym by open A.I. They provide a structured way to interface with data and test models and compare models across different environments. So, the suggestion here is that there might be the potential to use a similar concept to protect patient privacy, but still provide an interface to researchers who want to build new models. I'm not sure if any of our presenters would have a comment or anything to respond to that.

John Torous:
There are some ongoing efforts. I think York has one and University of Maryland, at least, are working on these things that are like data enclaves and this is a slightly different take on that. But how do you bring the researcher, vet the researcher, and bring the researcher to the data, rather than bring the data to the researcher? There's a lot of good paradigms there, and I'm happy to chat with anyone who is interested in that particular thing.

Jeffrey Girard:
I'll also say, if — as we're waiting for other questions to come in, we have a few other ones, too, that we have prepared just in case. So, one, I think, kind of common theme that emerged was this idea of trust and buy-in, both from the providers and the patients. And, you know, I think one of the things we had talked about was maybe trying to do a better job of aligning the types of metrics that, you know, doctors know and expect with maybe what is more common in the machine learning and predictive spaces. But what else do we need to do to make sure that, you know, trust and buy-in are optimized here and that we're really being responsive and kind of proactive about this? And so, this is a question for anybody that wants to chime in.

Male Speaker:
I'll be quick here. The users will tell you. This is at least in the app world, right? If you look at retention over time. So, John had that great slide about how quickly people are falling off, two weeks later they're not using most of these apps. That's a good indicator that that's not going to work over the long term to induce behavioral change. And so, we've done — we have our data helps, there's an opt-in study. We have 6,000 users that have been willing to be part of that for four years, and so part of that is we're listening with the [unintelligible] community or engaging with them. Not a focus group, but like continuously looking at retention rates. And so, like, we should take a page from the app world. They've done this really well because they're heavily incentivized to do it.

John Torous:
This is John. I think the trust issue is very important, because without trust you don't have health or mental health. I think the bar that we're asking for trust is that if people give data about their mental health it's used for their mental health, it's not going to be shared, traded, bartered. This is not an extremely high bar; we can go higher. But I think that if you kind of look at — unfortunately if you go to what consumers are using today, what patients and doctors are using today, it's such, kind of, an abysmal setting for trust and privacy that I think even small efforts — and we don't have to fix all of it — would kind of offer exponential increases in trust, which would lead to better engagement, to better quality outcomes. So, I think this is something we could fix today.

Jennifer Goldsack:
And John, if I could riff off that for just a quick second, as it packs the last question for our session. How important are things like, sort of, health and tech literacy in getting that right and getting that sort of communication and transparency, which is crucial to trust? Making sure that it's sort of equitable across the board?

John Torous:
We're kind of running our digital health training groups for patients with psychosis, we're actually very impressed with how quickly people can learn to use smart phones and they get excited about it. A lot of times, no one has actually sat down and said, "Did you know that your phone is probably tracking steps automatically?" Or, "Did you know here's how you can reach [unintelligible] medications automatically?"

These are things that people's phones can do. They're actually very excited and then they kind of want to build new skills and competencies, but I think, as we kind of move more digitally, first we're talking about here making sure people aren't digitally excluded because they don't have, again, the skills the competency and the confidence is probably of utmost importance. Again, it's not as fancy as some other things, but I think it's kind of the real work that we have to do to make sure that we help everyone.

Jennifer Goldsack:
Fantastic. And so, it looks like we're nearly at time. I want to say thank you to Jeff for being a brilliant co-moderator for this session. Also, to Glenn, to Connie, to Brian, John, Donelle, Matt, and James for being absolutely terrific, sort of, discussants for this session. You know, as we think about the sort of — the barriers and the challenges and the opportunities, what's clear to me after this conversation is that it's too simple to sort of point the finger at, you know, it's the tech that's the challenge or it's the sort of the human users or implementers that are holding things up.

It's really clear that, you know, there's issues really at the interface of the technology and of all of the humans involved. And really what shone through is the need to build trust, using evidence, and to do all of that built with sort of the end user, whether that's the clinician, whether that's the patient, or whether it's the person charged with sort of implementing this technology, keeping them front of mind as we build that trust and evidence. So, again, thank you so much to all of our experts. This was terrific. I learned a great deal and thank you for everyone for hanging on the line and for another great session, and I'll turn it back to our NIH — I think I got that right — hosts.

Sarah Lisanby:
Yes, thank you very much, Jenn and Jeff, for a stimulating session, which is a very nice segue into our last session prior to discussion, which will be on ethics, privacy, and special populations. Specifically, child/adolescent as well as geriatric patients as well. So, I'd like to thank our co-moderators for this session, Dr. Dilip Jeste, who is a senior associate dean for healthy aging and senior care, and distinguished professor of psychiatry and the Levi Memorial Chair in aging, and director of the Stein Institute for research on aging at UCSD. I'd also like to thank Dr. Jeremy Veenstra-Vanderweele, who is the [unintelligible] professor for the implementation of science for child/adolescent mental health at Columbia, and the director of the division of child and adolescent technology. And together, they are going to lead us through our discussion with these really outstanding panelists. So, let me hand it over to Dilip.

Dilip Jeste:
Thank you. Good afternoon, although it is still morning in California. Thank you, Holly, for inviting me to join this really important workshop. So, this last session on ethics, privacy, and special populations is especially important in the field of mental health. There are some special challenges in the field of psychology. One is that most of our outcome majors are subjective, unlike, say, radiology or oncology. Secondly, most of our patients and their families don't have access to state-of-the-art technology. And last but not least, our consumers are often vulnerable to being taken advantage of or when being abused by others. So, these issues are both privacy and security, as well as ethics, become critical. And I'm just delighted that we have a terrific panel of experts in various medical [spelled phonetically] areas, and we'll be hearing from them. Let me turn it over to my co-moderator, Jeremy.

Jeremy Veenstra-Vanderweele:
Thank you, Dilip. And I should say, it's been a really exciting workshop thus far. It feels like there's been a thread of ethical discussion running through from the very beginning. We are going to do our best to, at least for the first half, stay out of the way of our eminent group of panelists. Really a terrific group. We're going to provide only quite brief introductions and let them speak for themselves. And then in the second half, we will do our best to ask at least somewhat provocative questions before opening things up for a general discussion. Dilip, I think you'll introduce the first few?

Dilip Jeste:
Yes. So, I'm delighted to start with my colleague Camille Nebeker, who is associate professor of family medicine, public health, and psychiatry at University of California, San Diego. She's also director of ReCODE Health Research Center for Optimal/Digital Ethics. Camille?

Camille Nebeker:
Thank you, Dilip, thanks so much, and it's a pleasure. I wanted to start off, first of all, just to do a brief primer on what we mean when we talk about ethical, legal, and social implications. And I've just done this quickly so we can be grounded in at least how I'm thinking about ethical dimensions of research in clinical practice, and especially with respect to digital tools that are being used in mental health as well as general population health.

So, under the ethical domain, we think primarily of the Belmont principles of beneficence, respect, and justice, and is the consent accessible? Do the people that are — that we're asking to participate in studies or to use a certain type of clinical intervention, are they — do they have the health technology, data technology, technology literacy to be able to use whatever we're asking them to use?

Oftentimes, the technologies can pick up signals from people that are nearby, the participant or the patient, so we also need to think more about, what are the bystander rights with respect to using digital tools in health? Beneficence — the principle of beneficence is really about looking at the risks in relation to benefits. How do we identify these risks, quantify those risks, and then balance those in relation to potential benefits that could come to, not only the individual, but to the society at large? And justice is really about who is sharing the burden of any type of research we are pushing out? And are they like the people that are most in need of what's developed, what knowledge has developed? And so, access is a real critical part of that.

When we move into the regulatory domain, the focus is really on the law, which is not an area of specialization for me. But the federal regulations for research have been updated; do those updates really map to the types of activities that are happening in the digital health spectrum? We look at conflict of interest, the new privacy rules like GDPR and the California CCPR regulation.

And then social impact. We have to think about, what is the downstream effect of the things that we're deploying now? How have we engaged our communities in helping to determine whether or not the things that we're planning to do are right for them? And then this newer area, returning information, returning value to people that are participating in our study is becoming an area that is really important to unpack. Next slide, please.

So, think about the characteristics. These are also challenges of digital research. As we all know, we're very connected. The smart environment — we're using these tools more frequently in research. We can monitor and intervene with people 24/7 in real-time. We can capture just [inaudible] amounts of granular data. [inaudible] data anonymity anymore. We cannot even really be confident that deidentification is going to work, and so, you know, the other thing that's happened in the past six, seven years is that these tools have enabled access to people who are now doing studies on themselves.

So, a growing area of citizen science is happening. We also have, you know, our technology industries involved in biomedical research as well as non-profits. So, whereas we have regulations in research, for those of us in academic environments [inaudible] to accept federal funding, those regulations do not map to everyone who is involved in this ecosystem. And so, we have unregulated, regulated, untrained, trained, and people in this sector that are very thoughtful about ethical dimensions, and some that have no idea what they should be thinking about. And so, this is a really important area to be thoughtful of at the very front end of anything that's being developed to make sure that we don't experience those downstream effects. Next slide, please.

And so, in collaboration with Dr. John Torous and Rebecca Ellis, we have developed a framework that is created through an iterative research process, with ethical principles at the core. And what this is looking at is helping people who are developing technology, choosing technologies to use in research, clinicians that are trying to make a decision about what tool might be useful for their patients; but to think through the privacy implications, access and usability, data management, and risks and benefits. And so, what we've done is created, using this as our framework, a checklist that helps people to think through, "What do I need to think about before making a decision to use a certain product?" And so, I can talk more about that off the line and share this checklist and framework with those of you that are interested. And I'll pass it back to Dr. Jeste. Thank you.

Dilip Jeste:
Thank you, Camille. Our next speaker is Dr. Ken Duckworth, who is medical director of NAMI, the National Alliance on Mental Illness, the largest national grassroots organization in mental health. Ken?

Ken Duckworth:
Hello, Dilip, and hello everybody. I've learned a lot so far, and Dilip, I remember fondly a presentation at the NAMI convention some years back. We love our researchers. I just want to let you know, NAMI's had quite a couple of weeks. We're experiencing record calls at the help line and, you know, the state and local affiliates are being, I would say, overwhelmed with questions about mental health and COVID-19. One thing I want to offer from a few points of perspective is, when I talk to people at NAMI, and I do listen to them a lot, one of the problems we face as a group of individuals trying to make things better is they have a time [unintelligible] oversold by second generation [unintelligible] and psychotics that don't have side effects, for example, or for antidepressants that don't cause dry mouth. Each has been revelated to demonstrate different and profound problems for the population.

So, one of the things, you know, I'm interested in thinking about is how to frame this for people that it doesn't become another set of what people would say are disappointments after they've been told that new treatments aren't better — materially better. And I acknowledge that some new treatments are better for some people, but I just want you to know that's kind of heart of the framework with people. They're observing that they're taking the same medications or the mechanisms of actions as their family members might have two decades ago, and access to hospitals is hard, access to prisons is easy.

So, this is the — we have a loving and wonderful community, but a lot of people feel disaffected from the overall outcome of the fragmented, chaotic, patchwork mental health system. And so, how do you do co-creation with people with this experience? I heard several people talk about this idea, giving people feedback. How do you help people self-manage, ultimately, the idea that symptoms are useful for their own recovery journey? How do you do co-creation so that the people who are the end users of the process experience this as something that is collaborative and ultimately for them?

I do want to mention, on a different note, that NAMI did a project with Google putting up the PHQ-9 on mobile phones. And while I’m not at liberty to discuss the numbers of people who took the PHQ-9 on mobile phones, the number is enormous, and I feel like that's an example of technology that didn't require a lot of complexity, and we had large numbers of people come to the NAMI webpage as a result. In the British Medical Journal, I was invited to do a debate with a gentleman from England who had a much better accent than I did who was against this entire democratization of screening tools.

And so, it's just interesting to me. His concerns were largely about privacy — this is [unintelligible]. Privacy, and the lack of interpretation of these tools. At NAMI, we would trust people, you know, you have to empower people. They're already living with their symptoms, and we feel, like blood pressure, people should know their PHQ-9 score, to give one very simple example of how you can get technology out. So, until we have a uniform application of things like PHQ-9 in people's lexicon, like blood pressure 120 over 80, I think we are still going to face an uphill battle.

I was impressed by the critiques on privacy and the lack of trust of big data and institutions to guard individuals. Google has made it really clear that they had no access to anybody's information of any kind when they listed the PHQ-9. So, that was just an interesting, small experiment that we did. A lot of people took that and that didn't require a tremendous amount of collaborative co-creation because it's a tool that already existed. So, I'll stop there. And thank you all for all that you're doing, and I appreciate, you know, what I'm learning today.

Dilip Jeste:
Thank you, Ken. The next speaker is a colleague of Jeremy. Randy Auerbach is associate professor of psychiatry at Columbia. He's also director of translational research on effective disorders and suicide lab at Columbia. Randy?

Randy Auerbach:
Thank you for having me. So, in full disclosure, I'm not an ethicist by training, but I think deeply about these issues as it relates to trying to translate what we're doing in the lab to try to understand how we could actually detect it in real-time. And I think, perhaps most notably since my lab focuses on depression and suicide, we're really trying to figure out what and when. So, what are the things that actually — excuse me — are important to look at? And most notably, when do these things occur?

And — sorry, could you advance? And so, in addition to using kind of multi-modal neural imaging, we've also used passive sensor data collection through smart phones. And, as many of the speakers have explained in prior presentations, you know, this passive sensor technology, particularly as it relates to adolescents, provides untold access of what is going on in the contexts of adolescents' everyday life. And you could use a number of different sensors to kind of target very, very specific questions, and what we've done in our lab is really trying to characterize it as well as possible whether it be, you know, typically or neurally, and then trying to see if we could then kind of detect it and take it and translate it outside the lab.

And what's so promising about this in the context of adolescent samples — if you could advance the slide. Yeah, is that what you could do is then, it's not just kind of real-time assessment of these potential behaviors that may map on to depression or suicidal behaviors, but what it affords you is these just-in-time interventions. And what I'm meaning by this and what's been noted, kind of more colloquial throughout, is not necessarily intervening as in treatment, but intervening as an interceding when individuals may actually need services. So, perhaps the most [unintelligible] case is in the context of suicidal behaviors whereby you're able to intervene as somebody navigates into a risk state in the context of depression, and maybe interceding earlier in their disease course or earlier in the context of relapse when intervention may be more successful. So, it's really within this context and the context of adolescent populations that I've been thinking deeply about how to really protect our participants in ongoing studies throughout the lab. Next slide, please.

So, there are a wide range of issues at play here. The first being as kind of the access to private data, and so, when parents and teens come into the lab we actually explain in kind of painstaking detail as to kind of what we're collecting. And this what kind of circumnavigates the pictures that they're taking on their phone, the websites that they're trafficking, the words that they're writing in their phone, the locations and places that they're going to be, and we worked with our IRB to kind of figure out the appropriate language.

Remember, you have to kind of boil it down into a vernacular that is both kind of palatable to both adult populations but also adolescent populations, particularly adult populations where English may not be their native tongue. And so, what we've kind of done to ensure that there is kind of accessibility of this information is that we include a rudimentary quiz. You know, are they aware of the types of information that we're collecting, how long we're collecting, and things of that nature.

But you also see, in terms of looking at adolescents and adults, particularly parents, is that there's this disconnect in privacy. Generally speaking, adolescents are much more liberal in the type of information that they might be inclined to give away and parents tend to be more conservative. But it's also the case that, actually, parents aren't aware that they're currently giving a lot of this information away in terms of what they're doing on their phones and what apps they may or may not have downloaded and things of that nature. And so, it often begets just a broader conversation of privacy. And I think a point that was touched on by Camille much earlier is that, you know, can we guarantee privacy? And the answer is kind of categorically no. You know, we can do our best, we apply cutting edge encryption standards, we're updating those, you know, typically month by month as things improve. But for our purposes, we just try to acknowledge the limits of what we can and can't do.

Something that has been touched upon, I think, you know, briefly in probably each of the sections today is our inability in certain cases to do real-time data analysis. Now there are certainly studies that are able to do this, but when our prediction problem actually isn't solved — so, for example, Dr. Ryan did this predictions — these AUC findings at .92, but if we don't necessarily know the variables of interest going in but we have a thought of what might be of interest going in, you know, we really can't analyze.

And as I think some of the more — the engineers in today's presentation noted, it's actually fitting these data actually extremely complex, and so we both have kind of a what. You know, what are the variables that are of most interest? And then the how in terms of analyzing these data is extremely complex. And so, what we've done is we've really tried to just be forthright with [unintelligible] population, meaning that we view this as kind of a three step phase — at least three steps, the first being is that we're trying to identify high value targets across our different features within passive sensor data. Once we have identified some promising markers, we then seek to test these markers. And then as a third phase, you know, trying to validate it, which is not totally different than a lot of the RCT's that came before us.

And then, I think the last point from an ethical perspective, and I guess — this is not meant to be an exhaustive list, but something that is kind of active in the context of things that we do in the day-to-day is that there are these unanticipated real-time findings. An example that we could draw on from my own data is that we do a lot of natural language processing, and we are doing natural language processing, depending on the technique that you're employing, you find very interesting things when you analyze the words that people type in their phones.

So, you identify if things are unwanted sexual contact, molestation, sexual assault, and things of that nature. And particularly as it relates to teenagers, you don't necessarily always have the commensurate amount of information to necessarily file reports, and I think that we're grappling to try to figure out, what is the best way to handle these types of data in terms of protecting our patient population, protecting from perpetration of things like unwanted sexual contact. And again, what I encourage, and this is not the [unintelligible], this is that, you know, we consider these kind of active, ongoing consultations with our IRB to make sure that we're handling these issues in the most judicious and safe way as possible. Thank you.

Jeremy Veenstra-Vanderweele:
Thank you, Randy. These are difficult things to wrestle with. It's my pleasure to introduce Paresh Patel, who is an associate professor at the University of Michigan. We heard earlier from Peter Zandi about the National Network of Depression Centers, and Paresh is going to share with us some of the data use and management issues that have arisen regarding ethical issues within the mood outcomes program of the NNDC. Take it away, Paresh.

Paresh Patel:
Thank you, Jeremy. And thank you to Holly and NIH for their foresight in organizing this workshop. As Peter introduced in session two, the NNDC has a clinical and a research mission. One component of the latter is an awful large a rich registry of mood outcomes data in order to develop better treatment strategies for mood disorders. It's achieving this by cooperation among the 26 and growing academic centers and non-academic mental health treatment programs. We've already heard some of the co-considerations raised by other presenters in discussions this morning, and here I just want to highlight some of them that are relevant to the NNDC dataset and highlight on the slide.

It goes without saying that we're working with sensitive personal data and pains are taken to maintain privacy in the abstraction and storage of that data. The NNDC research data is considered secondary because it's derived from a primary dataset developed for providing just-in-time clinical decision support to clinicians, but it really challenges the ethical principal of respect and transparency because patients are not explicitly told about the deidentified abstraction, and are not aware other than in that sort of boilerplate language that most [unintelligible] have for general consent of treatment, that deidentified findings of their records will become part of a national registry. We can talk about this some more in the discussion, but this is an issue for all the [unintelligible] registries, many of which also collect PHI.

One of the biggest challenges to privacy is really sort of an embarrassment to our success, which is increasingly sophisticated analytic platforms, which could potentially reidentify patients to the linking of separate datasets. Luckily, some of the more fruitful discoveries come from a combination of datasets, but this is a privacy risk that's difficult to anticipate and arguably not universally considered by IRB's as they try to negotiate this complex and rapidly evolving landscape. Another ethical consideration with federated data is data integrity and accuracy. This is a principle that GDPR pushes.

People are mobile, and so federated clinical data may contain disparate fragments of a same individuals' data and the identified dataset that's very challenging. In our fragmented national healthcare system, that's even particularly challenging because we don't have universal patient identifiers, so communizing that data is really difficult and arguably almost impossible with deidentified datasets, but it introduces a lot of noise.

The ethical principle of justice is always challenging. Obviously, the ethical imperative to be attentive whose data we're leveraging, but most academic centers have developed data stewardship committees that hold the intellectual property in their clinical population. This was brought up earlier, and it raises the question that, as clinical margins to support research continue to shrink, what barriers might this intellectual property focus raise? Currently, in our experience, it hasn't been huge, but it does come up. A significant ethical concern with research on deidentified data is the beneficence for actionable findings. Deidentification make some research datasets possible, such as the NNDC research [inaudible], but it also creates a barrier to feedback for actionable findings. And it has been mentioned before, research populations often suffer from underrepresentation of disadvantaged populations are in collection reflects the underlying population, but it is biased by representation at our major academic centers.

Finally, I want to kind of raise a provocative quandary, which is not listed here but it's highlighted by the COVID epidemic. All of us in the clinical space are witness to what unprecedented relaxing of the privacy rules in order to address the acute threat that, by the latest estimates, could take the upwards of 100,000 lives in the U.S. And yet, these regulations present challenges to the kind of data we need to address the scourge of, for example, suicide death has been taking up to 50,000 lives annually for many years. I'm not advocating that we abrogate research related patient protections, but we really need to think about developing technologies and policies to balance the ethics of protecting patients and the ethical imperative, if you will, of quickly arriving at some more effective treatments, especially when it involves large-scale, multi-site, federated data. Next slide.

So, here, I just tried to think about some potential solutions, and these are not necessarily — I mean, not necessarily novel, and things that maybe people are already doing, like masking/coding sensitive information. Something that we, as a country, have not really come to grips with is a universal patient identifier, but it would really, potentially, significantly help towards integrity and accuracy in federated data. We need better mechanisms to re-contact patients of deidentified datasets for transparency and opt-out options and actionable findings, something like an Honest Broker system, which some institutions do deploy, but in the NNDC dataset that would be very challenging. There are some data management strategies that people are developing, some sophisticated algorithms for trying to manage PHI risks in distributed networks, with the idea that we need ways to make sure that integrated data doesn't accidentally reidentify subjects. As I mentioned, we need to promote models to balance the patients versus institutional rewards. Maybe increase incentives and decrease barriers for data sharing while maintaining safeguards for data security and patient confidentiality. And finally, figuring out incentives to balance the challenges recruiting underrepresented and disadvantaged populations. Thank you.

Jeremy Veenstra-Vanderweele:
Great. Thank you, Paresh, and thank you for keeping right to time. I'm conscious that we want to make sure that we do keep our next speakers to the four minutes each. It is my pleasure to introduce Luke Chang, assistance professor at Dartmouth, director of the computational social effective neuroscience laboratory, grappling with these questions in real-time.

Luke Chang:
Thanks. It's been an absolute pleasure to hear from all these different perspectives about the issues — the promise and also the issues of working also across different types of systems from private companies to healthcare networks and also from researchers. So, I think communal outlined a really nice ethical framework on how we should think about the different ethical principles for protecting privacy, but then also for maximizing benefits to our research participants. But I want to basically push on making a case for secondary data analysis, so these are basically when you might work on a dataset for a different question than it was originally developed for and what you had consent from participants from, and for what the funding might have been for. And there's a low of benefits for this, and that, you know, there's been a lot of conversations about this for the last few years, but I just want to highlight — some of the highlights.

So, one of them is that, by doing — enable the new research questions that weren't thought of in the beginning, and you can go back to old datasets and try to compare them or generate new things, and that — there's a lot of promise in that. It also allows us to enhance scientific reproducibility and minimizing false positives by basically increasing our power and sensitivity to [unintelligible] effects that might be there. And also, I think it's been mentioned by multiple people throughout the day, it facilitates development of new research methods and I think it might have been — I can't remember exactly who mentioned it this morning, but someone had mentioned some, like, image net and some of these datasets that have been publicly available for computer scientists to develop new models.

And a lot of the innovations that have come have been the result of, like, having better computation methods but also having much better data. And so, I think that's really important, and it's not always — those people are not always doing that work or not always on the teams of — that might have access to the data. It also reduces the costs of doing science if you can pool it, and then another kind of underappreciated thing, I think, is it protects these really valuable scientific resources that are generated from these datasets and an enormous amount of money that's poured into it. So, if a hard drive fails or a database gets corrupted or a researcher isn’t around anymore to share it, having redundant copies that are put out and stored around the world I think is really important. Next slide.

So, these are like the benefits of why we want to do secondary data analysis, but I think it raises even more of ethical issues. So, first, there's clear issues of privacy and, especially with HIPAA restricting anything that's identifiable. So, this could be face information if you're trying to get — detect, like, emotions, or signatures from voice about how someone's feeling that might put them at more greater risk for self-harm. These are — you can't really share these openly, easily. GPS location, for example, is another one where — and device identification. So, if you want to use, like, R.F. I.D.'s, radio frequency I.D.'s, to see, like, how many people are these people coming into contact with? Like, this type of stuff is impossible to share, even though it's being collected by all of these past incentive [spelled phonetically] studies.

Another thing is respect for persons. So, people give informed consent for the primary analysis, but there's all these questions that might come up and, in theory, they should be able to choose if they want their data used for these other things, but that's not always possible. They might not be able to be contacted, and also the people doing the secondary analyses might not be in touch with the ability to contact them. And there's been some legal cases about this that's raised a bunch of issues as well.

And then on the other side, you know, participants are spending all their time to participate in this, and we want to make sure that everything they do, they get the most benefit from even if it wasn't things that were initially intended from the original study. And then also, I think there's also — it can speak to justice where increasing the generalizability of our research findings and being able to train models that are much more robust and generalizable will make it accessible, hopefully, to many more populations who might not have been initially involved in some of the original data collection efforts.

Okay, so other issues that I think are ethical but also just really open questions, especially if datasets are kind of being collected across healthcare networks and private companies and also research institutes with federal funding. So, who owns the data? Is it the participants themselves, is it the researchers who collected the data, the research institutes, or corporations? Funding agencies who might have done it, or even the journals where it gets published? Like, these are things that are being grappled in many different fields. Sharing data is really important, but it's also incredibly costly.

So, it's time consuming, the amount of storage, especially when you start scaling up, is really large and the bandwidth to transfer datasets across people. There's some datasets in the [unintelligible] community which I'm more a part of where you actually have to mail hard drives because it's too big to be able to send over online. And also, I think an underappreciated thing is because there's these — you might get funding to collect data for a certain amount of time, but there's indefinite time horizons of what these responsibilities are on the people who've collected data share. So, this might persist for like decades, potentially, if you have a really interesting dataset. And that kind of ties into data stewardship.

So, often, you know, researchers are doing it, they have many different hats on that they're doing, and being administrator to sharing the data is a very arduous task and verifying that they have HIPAA compliance and all these other things, so that's expensive. There's also — might be legal expertise in negotiating license agreements and intellectual property and things like that. This is like, to some universities, charging handling fees upwards of thousands of dollars just to, like, handle these kinds of things.

Also, this could go on forever depending who collected the data, and also, who should shoulder this responsibility? Should it be the people who collected the data, should it be the persons themselves for funding? And then the last thing I want to say is, when you scale up the data sharing, there's lots of difficulties that emerge. And one of them that's been touched on, but I don't think it's been explicitly stated throughout the day, is the idea of data standardization. So, we can have all these data sets and maybe you can just download them, but if you don't know how to aggregate them or what all these variables are or who matches to who, that can be really onerous. Thanks.

Jeremy Veenstra-Vanderweele:
Thank you, Luke. And I'm privileged to introduce our last speaker who really has real-time experience managing found amounts of data. Acting director of the division of scientific programs for All of Us — the All of Us research program. Holly Garriock has her training in genetics and pharmacogenetics, which is one domain where we've actually seen these sorts of database decision making tools. [inaudible] for better or worse.

Holly Garriock:
Thank you, Jeremy. I'm going to do a quick check. Can you hear me okay?

Jeremy Veenstra-Vanderweele:
Yep, sounds great.

Holly Garriock:
Yeah, great. All right, so, I'm also not a card-carrying ethicist, but I do have our social/legal implications, or our ELSI specialist and lead for the program, she'll be on the call as well. So, if anything — any deep dive questions come in, I will call on her to help back me up. So, I hope here to be in line, largely, with a lot of the framework that Camille put forward in that first presentation, and I think what I'll be saying also addresses a lot of the points that Paresh made as well. So, hopefully this will be a nice flow of presentations from framework down to how we're actually doing it in this large national program.

So, hopefully I clearly — my slides got a little confused here, but hopefully everyone knows the All of Us research program is a large program aiming to enroll a million or more participants in the United States that are willing to donate their electronic health records, self-report survey questions, a fair amount of blood and urine, and a physical measurements assessment, as well as other mobile tech information over a longitudinal period of time.

We — this slide that I'm showing you here is how we try to balance a lot of our ethical/legal tensions that we have as we are implementing the program and partnering with our participants. The first one here is the appropriate reach. So, we have a large focus on diversity, and this goes beyond the kind of standard race/ethnicity definition of diversity. We have about nine industries of diversity that we use to account for underrepresented — you know, historically underrepresented in biomedical research.

And so, we want to make sure that we're not going to far, but we are going into the right places, both geographically as well as legally. So, enrolling the participants that we are able to enroll by law and groups that we're including and purposely not including at this point in time for ethical reasons. And then demographically, you know, as you know, I'm sure, the ages for consent vary by state, and so we're making sure that we are 18 and up in most states and 19 and up in some of the states where we have to be specific there.

This is a national implementation of that program, and so we're not able to bring participants into our office the way that Randy described and really sit down with them or their parents to talk about informed consent. So, this is an electronic consent for all of the participants that enroll. We need to make sure that the consent is at the correct reading level, we aim for fifth grade reading level or lower, and we need to make sure that they are engaging digitally in an appropriate way, and that our program really facilitates that. So, we've done a lot of work with sage bionetworks to try to make sure that happens. We include some formative assessments to try to promote understanding of the consent as they're going through it, and we also have a video support as well as onsite clinic support for those participants that need it.

For inclusivity, we are really trying to build trust in protecting our participants and our data. We largely do this through community engagement and ambassadors across the country. For maximizing benefit here, really want to focus on our concept around having data broadly available from community scientists to the individuals on this call, the academic ivory towers, and making sure that everybody has access to the data. We're building the infrastructure and hoping everyone will come into the data.

We do this using a cloud-based platform for broad access to try to get around all of the datasets and hard drive being passed across the country so all the researchers come to the cloud, analyze the data, use all the tools and everything there that's possible. In order to minimize harm, we do a lot of data transformation to try to prevent reidentification of the data. We have a couple tiers of access for data, we have a lot of policies and governance groups around — access to the data, access to the biospecimens, and access to the participants, and we have a lot of other policies that are in place to try to avoid stigmatization of data as well.

A lot more to say, I'm realizing I'm at my four minutes. I'm happy to leave the rest for questions. My next slide was around privacy, so it kind of shows the governance that we have in place at the program around privacy and security, and then the following slide is just around [unintelligible] information to participants, which is one of our core values and we really see as a return of value to participants. Sorry, Jeremy, went over.

Jeremy Veenstra-Vanderweele:
Thank you, Holly. No, appreciate you tracking your own time and holding to it, it's very helpful. I think recognizing the time, Dilip and I are going to go back and forth with some fairly rapid-fire questions but open them up to the overall group to chime in, and then also monitor questions in the background. Dilip, you want to ask the first question?

Dilip Jeste:
Thank you, Jeremy. Again, thank you all speakers, great talks. So, one theme it seemed was common to several of the presentations was giving feedback to the patients about research information we have collected. And it's not as simple a question as it may appear, because we are collecting information that is of research value, but does it have clinical value? And it may wind up going in opposite direction in the sense we may be interfering with the clinical management of the patient being done by somebody else. So, I just want to open this up for different speakers, what is your recommendation about giving feedback of the research information to the study participants?

Randy Auerbach:
This is Randy. I know it's an excellent question, and it's something that we grapple with. And briefly, I think it really depends on the clinical population with which you're working. You know, in our own data there's a real negative to have false positives for a suicide attempt. In so much as, you know, although, yes, ostensibly, you will keep this person safe, but, no, this person will likely not continue to participate in your work or clinically continue to use this tool.

And I think that Dr. Torous and several others intimated about kind of technology is great. Uptake is easy, but not incredibly easy. But dismissal of these tools when they're not working effectively is rapid. And I think that we need time, depending on the clinical question, to develop the right tools to address the right question.

Camille Nebeker:
I'd like to build on that, Randy. I think that this is such an important area. Return of information is not a traditional mindset of academics. And when it's clinically actionable, that's one area that, you know, in my, in my years as an IRB member we've struggled with. We've come up with protocols. How do you return information in a way that is responsible, that engages the clinician, and that doesn't insert upset?

And we're working with [unintelligible] on a longitudinal study with older adults. Everyone is over 65 years of age. And it's a five-year study where we're collecting physical data, mental health data, cognitive data, and also, they're using wearable sensors. So, we're getting so much information about them. And what we did is we wanted to know what is it you want back? How do you want to be engaged in the study for five years?

And this gets at short-term and long-term engagement. So, when we spoke to them and we conducted focus groups, we actually work closely with the people that are in the study as you do. And what we were saying to them is that, you know, we have this information. What we're trying to do is predict whether or not you have a higher likelihood of falling down six months from now, or that your, you know, cognitive abilities are declining. They said to us, "We want to know what you're learning. And we want to know not only how we compare with people that are like us, but how I compare to myself over time. " As Dilip said, some of this is clinically relevant and some of it is research data.

So, what we did, before we started to think about how do we return individual level data, we are looking at how do we return group level information. So, we took the publications that we'd put in peer reviewed literature, and we broke that information down. And we — you know, similar to what you'd see in a press. And we shared that with them. They said, "That's not quite good enough. We don't understand some of these words. We don't know how that related to us."

And then we took that information and synthesized it even more and created infographics. And we took it back to the community, and we said, "Well, what do you think about presenting information at a group level this way?" Got more feedback about words, about layout, about colors, and now we're on our third iteration. And I think engaging the community is critical for learning what we need to give back.

And it's basically tailoring it to them. Some want to know about actionable. Some people don't want to know if it's not actionable. So, we really have to be thoughtful. I think, you know, when I was working with the All of Us research program, it was returning assay information. Well, what if he find out that you live in an environment that is — you're exposed to lead. You're exposed to other toxins. You take that information to your physician; what can you do about it? And what — and then you get the question between the participant and their clinician. How did you get that information? What are you going to do with it? So, there's a big area of work that needs to be done in this area. So, I'll stop there.

Jeremy Veenstra-Vanderweele:
Yeah. Thank you. I wonder, Holly, if you could comment on this. I know return of information is a big issue in All of Us. If you could maybe talk about how you have the conversation with participants in advance about what they want to get back and not get back?

Holly Garriock:
Yeah, thank you, I would love to. So, our participants are partners. They're involved in many parts of our governance across the program and especially around return of information. And we really engage them as actively as possible.

So, our participants have the ability to choose which results they want to get back. So, we have a responsible return of genomic results. And there are different types of genomic results that you can give back to a participant. For each type of those type of results — medically actionable, traits, and ancestry are kind of the large three buckets that we work with. They are able to choose which kind of grouping they want to inform. So, we have a series of informing root loops within the consent process for return of information that the participants are able to engage in.

Not only do we have that kind of active online support, we also have a genomic counseling resource that we've put a lot of resources into. And both we have, you know, actual people on the phones that are there able to support them regardless of their genetic result. As well as the, you know, virtual experience that they're able to have.

Jeremy Veenstra-Vanderweele:
Thank you.

Holly Garriock:
But would you add anything there? Oh, Sorry.

Jeremy Veenstra-Vanderweele:
I don't know if she has access at the moment. I want to jump over to Paresh or Luke. Noting that there's also a question about whether people whose data is included in a large de-identified data set have some rights or should receive some information back if there's meaningful findings that come out of analysis there? Or if that simply belongs to the health system or someone else? I wonder if you could comment on that.

Paresh Patel:
Sure, this is Paresh. That is a really challenging question. I think, you know, it's easy for us to conceptualize what happens at the one end of the extreme. Where you have maybe the data from one individual that has a lot of scientific impact. I think Henrietta Lacks would be an example of that. And then the other end of the spectrum you have say Scandinavian healthcare systems, where when you query the data set you can literally query the entire population. Somewhere in between is where you have, you know, maybe five percent of the population really contributing to knowledge that helps the other 95 percent.

Is there a responsibility of you know returning some value to the individuals who participated? I think it's an unanswered question. And where that cut-off, if you will, occurs is really — I don't know. That's really something that we need help from ethicists on. And I think it's something that we often don't even think about with de-identified data. We sort of assumed, okay, we de-identified. The patient is no longer involved. And really, we have to think about there's some responsibility back to that patient.

Jeremy Veenstra-Vanderweele:
Yeah, I think you can think about some of the population level data, whole populations like Iceland who've participated in large research studies, and what that obligation is. It's a little different when you're part of an electronic medical record in a particular system.

We could pivot a little bit, and I'm thinking about some of the earlier conversations today. If we could maybe have a discussion of when you move from research to clinical implementation. And I think sometimes about the example of pharmacogenetics where we had what is essentially prospective retrospective, because genetics comes first, prediction of outcome in antidepressants and other treatment that rapidly turned into tests that were available for clinicians to use in everyday clinical care.

My experience as a clinician, mostly seeing patients and consultation, was it sometimes led clinicians to turn off their existing knowledge and defer to the results of the test. So, I work mostly with kids with autism. I'd see kids who are on a medicine that has no evidence whatsoever in autism but had a green light on the pharmacogenetic test result. And I think it's one of the things we have to keep in mind that just having a prediction doesn't mean that knowledge of the prediction will improve clinical care. And I wonder if folks could wrestle with that question a little bit?

Dilip Jeste:
You know, along that line actually it's continuing the same line of questioning. You can — I mean you have talked about the fact that the most important stakeholder for technology are the end users, the people who — the consumers. So, the question that I am asking is how do you see it from the perspective of the end users?

Jeremy Veenstra-Vanderweele:
You know, it's a great question, and I think there's a lot we don't know. I think the more the people we're trying to serve are engaged in the process. I love your earlier comment. You know, that when you're designing something, you're asking them in real time, "What would be useful for you?"

Because life is full of surprises. And some people may or may not want to schedule their appointments on their phone. They may be too paranoid for that. But some people will want that. I think it's going to differ by person. Anticipation of relapse will be important for some people, but I think ways to stay connected to others for people that are isolated, pre-viral isolation, you know due to their vulnerabilities, will be helpful for others.

So, I think this is a very interesting space. I think there is opportunity. And I salute those of you that are asking this question because I think this is the linchpin of this making a difference in people's lives. So, as you're asking the question, you're doing the thing that I'm really hopeful that you would be doing. So, I applaud your thinking.

Dilip Jeste:
Thank you. Anybody else has comment on that?

Sarah Lisanby:
I'm just going to make a general remark that we do have some other ethical experts who are not speakers but are participating online. We also have some Q and A from the chat box and Q and A box, which David Blightman is queuing up for us. When you're ready, just call in David, and then we'll bring that forward.

Randy Auerbach:
What is very common, I just wanted to say, and I think that this was highlighted very nicely in each one of the panels today, is that we kind of fall into this binary either/or situation. But I actually really do feel, and I think this was so well underscored in each panel, that it has to always be this kind of shared agreement between the clinician and the patient in terms of what tools are appropriate that is guided by kind of clinical insight and the patient's insight and relative experience. I don't think that these tools will ever exist in a vacuum, particularly as you think about moving from high income to low income countries, you know, from youth to adult populations who embrace technology in very different ways. So, I guess I just wanted to maybe make that comment.

Paresh Patel:
If I can just briefly comment on Jeremy's original question is how do we get that information back to the clinician and get the clinicians to use it? And that is, as Jeremy you're probably are very familiar with this, but those of us in clinical psychiatry have very — you know, there's very good data to establish that measurement based care is better than a standard or treatment as usual. And despite the strong evidence, it's very challenging to get clinicians to change that framework. Some data suggests it takes 17 years or something on that order to get people to change their practice. It's really like literally training a whole new group of people with the new skillset, but we've still got to keep doing it. I think it's really critical.

Jeremy Veenstra-Vanderweele:
Yeah, absolutely.

Holly Garriock:
In All of Us, I was going to say for when we're giving information value or results back to participants especially for the medically actionable genomic results, we try to make the reports as useful and responsibly interpreted for the providers as well. The participant has to bring the report to their provider. We don't provide information to their provider. So, we try to draw that line pretty starkly between research and medical care. But we try to make the information as accessible as possible for the participant to bring to their provider.

Jeremy Veenstra-Vanderweele:
That's a useful principle. Holly, I wonder if we could go to some of the questions from the chat box or the Q&A?

David Blightman:
Yes. So, this is David Blightman speaking. So, I've been looking at the chat box, and there's a question for Holly from Carol Ramos. "Holly mentioned oversight of how All of Us data are used. I'd be interested to hear more about that. And is there in particular — is there a particular framework they use for responsible data use? How do they weigh different values/perspectives of researchers/participants, et cetera?"

Holly Garriock:
We do have a lot of oversight not only of our users in terms of the different credentialing that has to happen for them to gain access to our different tiers of data but also to their behaviors in the research platform. So, we are able to see their workspaces and their peers are able to see their workspaces. And we publicly post the work that they do. So, it's a nice kind of auditing and self-auditing. But we also have groups within the program that are responsible for that. We also have the ability for the users of the data to be able to volunteer a review of their code and their workspace to see if it's potentially stigmatizing research, given our priority on underserved populations being represented in the data set.

David Blightman:
Okay, another question that arose the Q and A is a question from Tim Marianna who asks, "How do we handle the risk of unintentional bias in digital tools? For example, the machine learning algorithm may not by itself be biased but can provide biased output if trained on biased data."

Camille Nebeker:
I think that's just such a critical question because, you know, as most people know that the data that are used to train the machine learning process are creating the algorithms that are being deployed. And if the data set is not representative of the people that stand to benefit from the algorithm, the AI, that will be deployed, we're really doing an injustice.

So, I think about, you know, the use of the electronic medical record, and how it was designed for billing purposes. Now for some cases like radiology, ophthalmology, there's some really useful tools and uses of AI that make it a compliment to the physician's role. It is an assist.

When it comes to mental health, there's so much more that we need to know before we deploy AI and do it successfully. And in fact, one of the studies that I'm involved in was planning to use artificial intelligence to look at medication adherence, which in effect creates a weapon that could inadvertently discriminate against people that don't use their medications. And instead of looking at the social determinants of why they don't use their medications, it could be working two jobs. Having to get from one place to another, having — you know, just so many things in life that influence our behaviors. It's critical that the data sets that are being used really reflect the people that we're building them for.

David Blightman:
Does anybody else want to chime in on this question or I can go to the next question?

Luke Chang:
This is Luke speaking. The bias question I think is really interesting, and also, it's a really difficult thing to figure out what the right thing to do is. And I know it's certainly an active area in the machine learning community about how to deal with this. Because on the one hand you want to be able to — you know, we know from, you know, Paul Meehl and about, you know, actuarial versus clinical, you know, how we basically can make judgments, and how many dimensions of features we can actually use in humans. Even clinicians who have lots of expertise, they're still not using that many dimensions, maybe not more than like three simultaneously. Where we can have algorithms that can use many dimensions and also do really complicated things on how we extract features from that data.

So, on the one hand, if they can — if the goal is to predict something then we might not want them really muck around and muddle with the features because we don't really understand how everything interacts. But on the other hand, we might want to download certain things. Because in our cost function, while we might want to predict, we also might want to be minimizing bias. And we don't really necessarily have a good way to put down our cost function in how we're training the models themselves.

And then further, I think another thing that makes it difficult is multiple people throughout the day have brought up interpretability. So, on the one hand if we can see how a certain feature like a demographic characteristic or something like that is weighing into the model, then you can basically choose to ignore that, or, you know, when it gets really — if you're in these like deep learning things and you have all these non-linear complicated interactions with millions of parameters it becomes really difficult to figure out how to interpret it and how that bias is creeping into the model.

Female Speaker:
Thank you, Luke. I realize we are bumping in time, and we have the luxury of another hour for a broader discussion. Before handing things back to Dilip, I just want to say thank you to our panelists. Stimulating discussion, I especially appreciate folks acknowledging where there are questions that don't yet have answers.

Dilip Jeste:
Thank you on my behalf. Also, I think this because you've a great job. Thank you all for keeping your comments in time. So, it's one hour has been very productive. So, back to Holly. No.

Sarah Lisanby:
Okay. Well, thank you, Dilip. Thank you, Jeremy, for leading the discussion on this very important topic. So, the next hour is our general discussion. And over the course of the morning and early afternoon, we've heard from a broad range of researchers from a variety of disciplines — psychiatry, psychology, genetics, computer science, data science, engineering, just to name a few. We've heard from representatives from industry, including the electronic health record industry, technology and data companies, and pharma. We've heard from health systems like the VA. We've heard from SAMHSA. We've heard from professional organizations like American Psychiatric Association, nonprofit organizations, and patient advocacy.

And you've helped us in a short period of time get a broad overview of the landscape about our future vision of delivering effective and personalized mental healthcare for all that works and is equally accessible to all. You've helped us examine and take a closer look at existing networks and approaches to appreciate their strengths and challenges. And you've helped us map out what are the issues in getting from where we are today to that envisioned future with a very nice discussion about how technology they not be the major challenge. We also have to think about culture, incentives, ethics, and the human factors.

To cue up our general discussion, I would like to go back to something that Ken Duckworth said, which I think is really worth underscoring. He told us that the NAMI call lines are being overwhelmed with people calling with mental health concerns in the midst of the COVID-19 pandemic. And he — that's the — that's our current reality.

And he pointed out with that as a backdrop that patients are not well served by fragmentation in the mental health system. He reminded us that the patients that we serve are disappointed by prior treatments that over-promised and under-delivered.

So, with that in mind, I'd like to open this up to a general discussion. And I'd like to ask each of us to focus on what do we need to put in place concretely to be able to build towards a future where multimodal data can inform treatment selection for mental health in a way that's accessible and effective for all. And going back to the vision that Dr. Gordon presented to us in the morning, thinking about what role we can each play in achieving that future.

The panelists are welcome to unmute themselves and participate. You're also welcome to use the chat boxes. And I've also asked our Q and A moderators from each of the sessions who had leftover questions to bring them forward so that we can have a fuller discussion. So, I’d like to open it up there for discussion amongst the group.

Greg Simon:
Yeah. This is Greg Simon. I've unmuted myself. I think you can hear me.

Sarah Lisanby:

Greg Simon:
You know, I'll get on one of my soapboxes that some of you may have heard me climb on before. But, you know, to me it's really essential that we distinguish between, you know, privacy issues and proprietary issues. And my experience is that those sometimes get blurred to serve the interests of, you know, particular stakeholders who, you know, basically would like — and it tends to be more proprietary stakeholders.

So, you know, one of the questions that came up earlier in the discussion is who owns the data? And at least, you know, in my network or in the world I operate in we are very clear that health system members own the data. You know, there's no question about that. That's a legal principle that, and it's also a foundational ethical principle that that health system members own the data. They are their data. We borrow those data temporarily to learn from them. We are guests in that house. And that we have to behave as guests and always respect people's rights and try to communicate with them clearly about how, when, and where their data will be used. So, that's pretty foundational.

But then on top of that, when we talk about sort of the proprietary interests in what is learned from those data, I feel pretty strongly that that's public domain work. And if our members of these large healthcare systems have said yes, you know, you may use the data that we contribute for research. And when we've talked to people about this, they are pretty clear that they feel a strong obligation to help other people who might also suffer from mental health conditions. But they have strong objections to that knowledge being privatized.

So, I think, you know, we need a pretty clear distinction between there. And you know, I think in terms of, you know, creating the incentives for, you know, privacy protection and public domain learning. Or you know, as I sometimes say, if your code and all your results aren't on GitHub you should say it never happened. That's my soapbox. I'll climb down now.

Sarah Lisanby:
Thank you for raising that and the topic of balancing access and open source. Not only with respect to the data but also the algorithms that are published from the data, balancing that with the privacy issues.

And you raise another aspect, proprietary nature. We do have representatives from different sectors here, industry as well as academia. And I'd love to hear some discussion. Because I think depending on where you sit, it influences what you experience on that decline. Just in terms of accelerating science, we think that open source data and open source algorithms. Using things like depositing your data in a national data occupy, for example, these can be research accelerators. But there may be incentives that — or disincentives, let's say from people participating in that. Can we have any comments on this issue?

Greg Simon:
Yeah, well, this is Greg. If I could just add one comment because I think there clearly are privacy issues related to data sharing. Which our group is way into the weeds on and happy to talk more about. But it gets very nerdy. But there are no privacy issues related to sharing of code. You know? So, for us patient's data are behind a pretty high wall, and we have to be very careful. But every line of code we have ever written is on GitHub.

Sarah Lisanby:
And may I ask the researchers here is do you agree with those comments? Or is Greg the outlier here?

Luke Chang:
I'm happy to fully support Greg's sharing of code on GitHub. Or at least the code. The data, I think, is really complicated. I think — on the one hand I think — I really appreciate his perspective that he — where the patients own the data or the network owns the data. And you know, it's the privilege of everyone who has to work with it.

But then on the other hand, there's a bunch of things — when you start adding — this is more from my end user, not so much on my generating hat. When there's really large barriers to accessing the data, it makes it — so, for example, even, you know, the NIH's data sharing. It's great that they're putting all this effort into it. But then on the other hand it's — in practice, I find it really hard to — it's very hard to put data in, and it's really hard to get data out of it.

And other things. Like I've waited a year sometimes to be able to get access to data sets. And then I've been rejected and rejected, or you have to jump through so many hoops. You have to get a legal team in. And not everyone has the resources to be able to do that. So, then it's not really democratizing or whatever, however you pronounce the word, access to the data and the types of questions and things they can use with it. So, I feel mixed about it, but I fully appreciate what Greg's been doing.

Sarah Lisanby:
So, I think you're making the point that it's not just about researchers being willing to share their data, it's about making it easier to access, lowering the barriers for investigators to be able to use that data. Any thoughts on this topic, data sharing and code sharing?

Greg Simon:
This is Greg. I was going to add, you know, I am a big fan of what All of Us is doing in this space. And I wanted to sort of go back and underline something that was talked about related to All of Us. Which I think is going to be a really big issue for the research community to think of.

You know, it's that the specific sort of you know, All of Us idea that you can have access to these data, but you need to operate in public. Meaning, you know, Jupiter notebooks, I'm getting nerdy about that or whatever system you're using to access those data, your code are available for public inspection. So, that people can make sure you're not doing bad things.

What that means is that some of researchers' proprietary interests — you know, I'm able to ask questions privately and keep the answers private to myself until I'm ready to release them — have to go away. That would certainly accelerate learning and would break down silos. But it's not — it has not been part of our culture to say that everyone in the world can see the results of my analyses as soon as I can see them. That would be a radical change in how we do business.

Sarah Lisanby:
And that's another aspect or element of trust when you're talking about do people who provide the data trust the systems to keep it private? There's also another trust issue of science. Do we trust the researchers or the scientists to be transparent about how they're using data, how they're generating those results? These are fundamental issues that are key to reproducibility and rigor in science. It's something that we definitely care a lot about at NIH for us to trust [inaudible].

And let me just remind you as you want to make comments, please unmute yourselves. And we do have Adam Thomas who is monitoring the chat box and Q and A box, if you want to make comments there.

Jeffery Girard:
Yeah, if I could just — this is Jeff Girard here. If I could weigh in on Greg's point about the sharing of code. I think, you know, as a general principle, you know, I fully endorse that. But I do think that there's, there's nuance here of course, which he alluded to. I think there's different reasons to share code, different purposes for doing so.

So, there's probably a continuum between, you know, perhaps sharing your code in order to, you know, yeah, just show, "Hey, these are the analyses I did. You can look and check for typos or bugs." And it's much more about that reproducibility, transparency piece.

Hopefully there's very little kind of arguments against that. It's hard to imagine what the downside of that would be. I think the — but there's also — on this continuum there's also the purposes of sharing for like, let's say, developing software for other people to use, right? So, let's say we want to implement this model that's going to, like in the type of work that I do, will detect facial expressions. Now, other people are going to download the software, use it on their data, and I think that that has the potential to be an incredible accelerant to work.

But it also — I think with that great power comes great responsibility to make sure that if you're going to make something easy, that we're accelerating us on the right path. That we're not going to charge down on something that's poorly validated or not working that is now going to sort of balloon into this whole area that's not very trustworthy. So, I do think that, yeah, like the more you expect people to reuse your code, the more effort you need to put into validating it and, you know, commenting it and documenting it, things like that. So, that's an important, I think, note to add.

Sarah Lisanby:
Yeah, thank you for sharing that. And I know that we have a computer scientists and more professional software developers who are participating. And I think it might be worth discussing what sort of collaborations are needed across disciplines from the clinical sphere clinical research collaborating with the computer scientists or software developers or the engineering side to be able to be sure that those software tools are well-documented, able to be used by others, don't break when the operating system changes, and are supported. And it's a big deal to support software. I mean, typically companies support software.

But there are open source examples of — like Linux or things that are — where the package is open source. Can I hear some discussion across these disciplines about what sort of collaborations are needed?

Male Speaker:

Jeffery Girard:
Sorry, I'll just add really quickly that, before we go on, I think the collaboration piece is an excellent one. I think another thing is funding. That a lot of those open source software packages are really being supported with people working nights and weekends. Like blood, sweat, and tears put into these things. And I just think there's not enough funding for that type of really critical work. So, but also the collaboration piece.

Holly Garriock:
Was that, Guillermo? Someone else wanted to speak.

Guillermo Sapiro:
Yes, it is, thanks. Thanks, Holly. I just wanted to make a quick comment about sharing software. And if there is multiple — as what to said — there's multiple reasons. One of the ways to share software and data without sharing software or data is through cloud computing, something that we are exploring, and Duke and other universities are connecting their health system with the cloud — PHI, HIPAA, everything.

So, that will allow you at any institution to run my software without me having to share the software with you. And will also allow me to test on your data without you having to share the data explicitly with me. So, I just wanted from the technology perspective that a lot of the concerns that we are having are going to be improved with the use of cloud computing.

So, security, transparency, who touched the data, who touched the code? So, I'm basically — that's actually the meeting I'm jumping in the next 15 minutes. So, I just wanted the non-technology people to be aware that there is technology, not to address everything, but, yes, to address some of these sharing — both of software and of data.

And, as I said, there is multiple institutions that are leading and paving the way for that. And I would be happy if somebody wants to know more to contact me offline. But it's not just the old fashion that I have to send you my code that you have to install. Many institutions and many companies are working very hard to make a standardization of this software for healthcare through cloud computing.

Sarah Lisanby:
Thank you, Guillermo. Those are very important points. And this might be an example where there are already existing technological solutions. And the problem is not the technology, per se, but the incentives that make the individual stakeholders unwilling to use those solutions because of wanting to keep things proprietary, either for research reasons or for reasons that their own industry financial interests.

So, I'm wondering if — we've been talking about data sharing, and I know we do still have our colleagues from Epic on the line. When we think about the clinical data and [inaudible] regarding privacy of clinical data within our electronic health record system. And of course, we have multiple electronic health records systems. So, it's a bit of a patchwork of different systems that are in use in across the country and across the world.

I'm wondering — I'd love if our Epic colleagues are still on the line might want to address how they view this in terms of the idea of how we might share between electronic health records systems? Or our colleagues from the NHRN or other [inaudible] networks that have addressed interoperability between and across EHR systems.

Matt Switzler:
Yeah, so, I'll be the Epic guy on the line. But, yeah, I think there are some well-established standards for exchanging information between health systems. Those kind of fell out of the meaningful use initiatives. One barrier I see the doing that is that a lot of behavioral health providers were actually exempt from those meaningful use and EHR incentive programs. So, in our experience there still is a lot of faxing and phone calls and mailing to work with some of those organizations. So, I feel like maybe from a national level getting some incentives there might be useful.

Another sort of thing that I'm thinking through that we need to be thoughtful of is interfaces are great. But I think we need to be thoughtful not to have, you know, 13 or 14 different standards that folks are programming to because then all of a sudden, your EHR vendors are going to be just programming interfaces all day. So, I think one area that we use help from an industry perspective on is getting sponsorship of interface standards and holding different software vendors to those standards, which is kind of a tricky thing to do, especially as this is a rapidly evolving field. But we will need interfaces so that we don't have to reinvent the wheel if organizations want to plug and play with data. So, that's kind of my thoughts on that one.

Sarah Lisanby:
So, thank you for that. So, you're pointing out that meaningful use was meant to facilitate the sharing of health data as people move through different health systems. But you come into that for behavioral health there's exemption. And you commented that there might need to be some national incentives to overcome that. And sponsorship of interface standards, holding software vendors to those standards, can you say more about — for those of us who are not in the software and computer science space about this, what you commented, sponsorship of interface standards with software vendors? What would that look like? And in whose domain is that?

Matt Switzler:
I think it falls somewhere in the software space in terms of who defines standard. You can imagine two computer systems talking to each other, and they need to be speaking the language to have a meaningful interaction. However, if you're expecting sort of one system to speak a hundred different languages that's hard to implement and support from a technical perspective. Whereas if everyone was speaking the same language the data flows much more smoothly. So, that's kind of what I'm referring to when we talk about interface standards.

Part of that is a common sort of dictionary of data elements. For traditional medical information, like medications or diagnoses, we have things like RX Norm and ICD10 and that kind of provides those standardized code sets. But for a lot of the sets of information that behavioral healthcare providers care about, things like questionnaires or other social factors, those data dictionaries don't necessarily exist yet. So, I think one thing that we could get help from groups like NIMH would be to have those standard data dictionaries. Because then everyone's programmed to the same standard.

Sarah Lisanby:
Great, thank you for that. And I'd love to hear from — a lot of our investigators who spoke earlier commented on the strategies they've taken to try to interface with the EHR and use it for improving mental health outcomes. So, I'd love to hear some discussion between these groups about where the potentials are.

Diana Clark:
Hi, this is Diana Clark from the APA.

Sarah Lisanby:
Thank you, go ahead.

Diana Clark:
I see — I think one of the things that we have actually — and it's something we've actually sat down and talked to NIMH about as well. It's this idea of actually, I agree, identifying common data elements that can be used across the different home health systems and the different EHRs. And I mean that is going to be extremely important for us to have comparable data across the different systems.

And another way to do this, too, is — in addition to identifying common data elements another thing that could be done is to — even if different HRS and different systems are using, say for example, different patient reported outcome measure for the same outcome, for example, depression. If you can figure out ways to create a crosswalk between those, so that even if one system isn't — you know and limit those tools.

So, even if I'm one system is using, for example, the PHQ9 and another system is using for the GDS — because we know that for geriatric patients the GDS tend to work better for them as opposed to the PHQ9. So, something like that if we can figure out a way to make that crosswalk, that would be really important.

And that's something that I know we at APA we're trying to work towards. And for example, working with a team from HRQ team, H-R-Q, to actually try to see how can we harmonize depressive measures across different registries. For example, ours as well as the prime registry and then across different health systems.

So, it's something — it's — we're taking steps towards that. But, definitely, I agree with on the Epic person that we have to have common data elements, define the common standards, in order for us to have data that can be comparable.

Sarah Lisanby:
Great, thank you, those are very important comments. So, it's not just the software interface, but it's also the measures themselves. And having crosswalks across the measures so that we can be able to look at the same thing, whether it's a measurement of an outcome or a clinical feature, across systems. Because — and it was in the earlier — nicely mapped out some of the challenges that researchers face. When you're trying to do research in the real world, you're having to make use of what is actually there. And it may be a different type of thing that you would have measured in your conventional clinical trial type of setting.

Paresh Patel:
Holly, if I can raise a point, and this might be a good point to intervene on that, there's an, as someone mentioned earlier, there's an engineering problem, but there's also a sort of a social-cultural problem. And I think this is something that maybe Matt from Epic, you can speak to if you have enough line of sight to it. But it's something like Cosmos which is, you know, Epic's attempt at a very large sort of common data model structure. There are challenges there for using it for research.

And I don't know if, Matt, if you have any line of sight? What are the social cultural challenges to using that data for research? Because clearly that data can be used for clinical purposes, but the rules of the road limit how it can be used. And maybe, you know, just speak to some of the barriers to using that kind of data for research.

Matt Switzler:
That might've been my phone, sorry, everybody. Paresh, I probably don't have all of the insight into the Cosmos data set and some of the challenges there. I do think there is an opportunity there for either usage of the Cosmos data set or a similar model. Kind of the idea of Cosmos is that we use the same interface standards that EHRs used to talk to each other to pull into a common medical data set that Epic organizations have access to. And the benefit there is that we don't — we sort of have that common language and that common use of that data for organizations to use. And we can do some of this more advanced research without having to worry about sort of the data translation components of it.

So, that's something we're working on. I'm not sure, Paresh, if you had sort of more targeted questions for me on that one? I know I'm not the Cosmos expert here, but.

Paresh Patel:
Sorry, I hope I'm not causing the feedback.

Adam Thomas:
Sorry, I think that is coming off your line, Paresh. This is Adam. I'm hosting. I just hit mute all everybody. Try again, Paresh.

Paresh Patel:
Okay, now, okay?

Adam Thomas:
It's okay.

Paresh Patel:
I was just speaking to, you know, early on — and this is — I think this is why I didn't want this discussion to be Epic-centric. Early on when we were looking at things like cosmos, which are really large federated data sets, it was, you know, the challenge is institutional resistance to sharing data for research. And I don't know how that's evolved over time, but it's one of those — it speaks to how hard it is to get that data together across multiple centers, and that maybe there are opportunities there for developing a common framework for sharing that data.

Sarah Lisanby:
You know, one thing I wanted to bring up is what is in the data set to begin with, and what is actually in the EHR? When we think about mental health treatment, it's not just about medications. It's also about psychotherapy and behavioral interventions. And earlier in the day, we heard comment that we might know that a psychotherapy session occurred or was paid for, but we might not have in the EHR information about what type of therapy or the quality of the therapy. And I'm wondering if any of our colleagues on the line who may have a background in this have any comments about what we — should we be capturing something more about psychotherapy and cognitive behavioral interventions?

Greg Simon:
Yeah, this is Greg Simon. I was going to say I think this is a really critical issue, and it's a real challenge. You know, and I think we probably need to consider it at multiple levels. You know, one of those is how can we at least get providers to record in a more consistent way? Specific sort of families or schools of therapy or therapeutic interventions to at least report those in some standard way. And work with the EHR vendors, especially specialty mental health EHR vendors, about, you know, trying to people to record that more systematically. Eventually, it may be payers who say, "You really have to record those data, you know, for payment." That may drive that.

Another one that is very relevant this month is, you know, in many of our healthcare systems mental health care is switching to virtual care. And, you know, many mental health providers are now using various technology platforms to interact via video conference. That — and I suspect some of that will stick after we get past this crisis. There's an interesting question there because what it means is that now the actual content — the content, the lexical content, that tone of voice content, and even all the video content of psychotherapy — is now passing through someone's pipe when it used to occur completely in private.

There is an interesting, fantastic, and also somewhat frightening possibility there of how we would mine those data. And you know, I have colleagues who are very interested in those questions about how we might be able to mine the lexical content, the tone of voice content, and facial expression content from video psychotherapy sessions to learn about what actually happened. It could be a relatively dramatic advance in understanding how and why psychotherapy works, but there are a lot of issues to overcome about privacy. So, that's a — but I think we have moved so far to virtual care in the last month that we will probably pull back from that somewhat, but I think it's going to be a much bigger part of our future.

Sarah Lisanby:
Great, thanks for making that point. And it was referred to earlier in our session as the three V's of telemedicine, which I had not heard it referred to that way before, but the verbal, the vocal, and the visual. And as we are doing this conference right now, a virtual conference, we are communicating in those three ways. And the degree to which that sort of information can be used to improve the health outcomes is certainly an exciting topic to explore. Raising not only ethical and privacy issues but also technical issues of how you handle that data and what it could mean. Any, any thoughts on that? I know we've got some real deep digital phenotyping experts who look at these devices and other types of approaches.

Male Speaker:
I had to mute all again. Please go, thank you.

Diana Clarke:
Okay, this is Diana from the APA. So, one of the things that I — around the idea of online psychotherapy, I think, we could probably learn from the Australians with that. And, you know, so just to kind of reach out to see what are some of the results they've had. Because since 2012 do know that Gavin Andrews and his group in Australia have been doing online psychotherapy and with video conferences. So, we could potentially learn from them as opposed to, you know, starting from scratch. Then so that's one thing.

And then, the next piece I was going to actually — oh, my goodness. It's slipped from my brain. I'll remember, and I'll come back to it. It's gone. Sorry, I guess I need another cup of coffee. The second thing I was going to say is completely gone. But we can actually learn from some of our colleagues in other parts of the world.

Sarah Lisanby:
Yeah, so, thank you for that point in terms of the telemedicine, tele-psychiatry, online psychotherapy going on. And I think the issue that was being raised was how to capture that data. How useful could it be? And how, if it's being captured, it might change the therapeutic interaction? Any discussion on that?

Diana Clarke:
Sorry, I'm going to go back to the issue that I was, I wanted to really talk about, but then I jumped in and mentioned that. I agree with Greg that we do need for a standardized way that clinician, you know, the clinician to document things. But I really do believe, and this is something that we hear from clinician all the time, is that we have to really make sure we balance the clinicians clicking off a box and selecting something versus having the ability to write their notes and, you know, free form notes. So, we have to balance those two things.

So, that's — I wanted to say that first because that's really important. And I know that's where you do find clinician opting out of certain things. Because they find that they're just having to click all these boxes. And it's supposedly interfering with their ability to do the clinical work that they're supposed to do. So, we need to balance those two things.

Sarah Lisanby:
I'm really glad you pointed that out. We've talked some about patient burden, but what you're bringing up now is provider burden. And when we consider the pressures on providers these days, as many of you will know, if it doesn't help the provider be more efficient and take better care of the patient it's probably not going to happen because of the time pressures. So, delivering value to the clinician is clearly very important and being able to do that in a way that does not add to the burden of documentation.

Peter Zandi:
Hello, this is Peter. I'll just speak to that a little bit, because we were doing that with the — am I on mute? No, sorry. We're doing that with the ECT clinics in the NNDC where we're trying to encourage harmonized documentation of ECT procedure. And in doing that, we are trying to focus on two goals. It being clinically useful to the providers and to the patients and to the actual clinical encounter. And at the same time collecting structured and harmonized data from that ECT procedure that will be useful for the sort of research and quality improvement activities down the line.

And I think that same sort of thinking can be applied to psychotherapy or other types of procedures. And I think it is a worthwhile thing to think about. So, it isn't what we think about when we're dealing with EMR data where it's garbage in, garbage out. But we're getting good data that's not only good for clinical purposes but also for research purposes.

I will say it's a lot of work. I spent a lot of time with the clinicians trying to get them to think through what would be useful in the of documenting these procedures for them clinically but also can be captured in a way that will be useful for research purposes. It's just a lot [inaudible].

Sarah Lisanby:
Thank you for bringing that up here. And when we think about how we can use these streams of data, including the EHR, to inform treatment selection, we're discussing now do we have adequate information in the electronic health record to be able to make that — to inform that treatment decision? In the case of medications, it is common that you have the medication name and the dosage milligrams. But in the case of procedures, like ECT or TMS or psychotherapy, you, you might have an indication that the procedure was done, but you might not have the dosage or enough information about that procedure to be able to have the system learn how best and to whom to use it. Because it's like writing down, "I prescribed Prozac,” but not bothering to mention what dosage you prescribed. And then, yeah, that's just not enough information to inform that.

So, in thinking about what things we need to put in place concretely to build towards a future where multiple data can inform treatment selection for mental health — this is one of the concrete things you're illustrating Peter. Building into the EHR, sufficient resolution and documentation of the procedures that we're actually doing, and you're ECT example is one very good example of having done that. May I ask you did you build that into Epic, and is that something that can be shared? And do you see this becoming a standard on how to document the ECT procedure?

Peter Zandi:
That's the hope actually, yes. We're working with all our clinicians across the NNDC to come up with recommendations about what would be optimal for clinical purposes and hopefully for research down the line about these procedures. And we hope to publish a paper to report on those findings and also to create tools — particularly in Epic, whether it's a flow sheet or a smart form — that we can provide to other clinicians so that they can easily adopt these flow sheets as a tool, a document, for ECT in their clinics. So, the hope is that we can advocate for this becoming a standard way of documenting these procedures and spreading it. And not just within the NNDC, but beyond as well to move towards a more standardized approach of capturing the structured information.

And while we're doing this, I'll just go back to the point that Diana made that I do think it is important to balance without doing this in such a way that doesn't burden the provider. And the provider still wants to document in their notes a lot of information. So, we're trying to balance in good structured information with the flexibility to add details in their notes. And, again, yes, we're trying to disseminate this in a way that would become more standard for other clinics to use as well.

Sarah Lisanby:
Thank you. So, we're thinking about what concretely needs to be put in place to accomplish this goal. We've talked about a number of different the factors. One being the literal software standards for interface across EHR systems. We've talked about the need for data dictionaries and common data elements, the need to balance the provider burden so that the thing will actually be collected on for it to be feasible. We've talked about open source sharing of the data and the algorithms and removing barriers to accessing the data and that and the algorithms. And right now, we're talking about building up the dosage information, so that we have enough information regarding the interventions that are given so that we know — so that we can link those to response.

Are there — what else do we need to put in place? What needs to be built concretely to put in the framework where an AI ecosystem could be envisioned in the next five, 10, 20 years that would become part of a learning health system for mental health specifically?

Peter Zandi:
I'll just — this is Peter again. Just two comments to build off of what was already brought up by a previous folks. One was that I'd endorse the idea by Diana, work to establish how to do a crosswalk between different patient reported outcome measures. That's a really interesting idea. Because we do find different centers like to use different measures, and it would be good to allow them to continue to do what they like but allow us to bring that data together and use it in a larger data set, even though they do have different measures. So, I think that's something that could be encouraged. Work that tries to evaluate and establish psychometric crosswalks between these different measures.

And then a second point that I think would be useful in terms of concrete steps. I'm not quite sure how it works. This builds on the point that Matt said. A lot of these patient reported outcome measures, as he mentioned, they're captured in Epic. But they're not — there's not a standard coding system for them. So, they live in these flow sheets or smart forms. And each institution captures them in a different way.

So, even within systems that use Epic, trying to bring that data together on the PHQ9 measures might be a little bit challenging; not impossible, but it's a lot of leg work. But it would be made a lot easier if there was some standard nomenclature for pulling that data. And I actually don't know what would be the way to drive that to happen. How do we — how do we incentivize adoption of new cloning systems for these other sorts of things that we don't have codes for, like low-inc, for example? [Inaudible] curious, is there a way to encourage or incentivize that?

Greg Simon:
Yeah, this — Greg, I was just coming on that point about standard nomenclatures for patient-reported outcomes. It is an issue, and there is nothing equivalent to low-inc. in our network, we have sort of adopted a standard nomenclature for representing PRO data that I think is relatively robust because it allows you to capture, you know, actual wording of questionnaires, motive administration, things like that.

It's a public domain. You know, it's a — it's a public-domain data model. We put it out there for anyone to use. And we have also built a sort of crawler tool that will troll through the backend of anyone's EPC database, since our systems use EPC, and find all the instances of common mental health patient-reported outcomes by searching texts of labels, questionnaire items, and so on. All of that stuff, you know, is free to share. I mean, we are not a standard-setting organization, so we — it's not like we say other people should use that. But we say, "We've got some stuff that we think works pretty well for us." And you can have it for free; just contact me.

Peter Zandi:
That's great, Greg. Actually, I didn't know about that, and I will be contacting you about that.

Greg Simon:

Sarah Lisanby:
So, I'd like to do a quick time check. We have about 15 minutes left, and I'm really thrilled to see that we still have almost 100 participants online. You really stuck with us, and I believe that's a testament to the level of interest that there is in the community about these issues. I want to be sure that we cast I wide net here. I know that, Adam, you've been taking the questions from the chat boxes. I want to let you bring forward any questions that haven’t been aired yet. And also, other participants that we haven't heard from yet, want to be sure that, if you've been waiting for a break in the conversation, to be able to make your points, that you feel welcome to do so now.

Adam Thomas:
Thanks, Holly. There hasn't been a lot of chat — questions recently, but I would like to take the opportunity to pull in one from earlier this morning, that came from Javier Gonzalez-Castillo [spelled phonetically]. And I — actually, I might add my own spin. So, he noted that there was some discussion of wearable technologies, and he was wondering about priorities. People could elaborate on what they see as the most important and high-priority data are to collect. But I would like to expand that a little bit. I — to suggest, like, what other priorities could we add? So, that would include genomics; that would include other biomarkers. What do the folks on the clinical frontlines think that the larger categories of data are, that we're missing, that could really increase our prediction quality?

Susan Lisanby:
Please unmute yourself if you'd like to respond.

Peter Zandi:
This is Peter again. Sorry, I'll just jump in with a real quick comment here, just real quickly, that not so much a specific category of data, but what's interesting: There are a lot of these — the general censor-type data that we've been talking about, digital phenotype data, whether it's mobile or wearable technology. What I think — but others can correct me if I'm wrong — where there's a DRTHA [spelled phonetically] — is connecting that digital phenotype data with the electronic medical record data to bring these two strands together to see how they correlate with one another and inform each other. So, I do think that should be an area of active investigation: how to bring those two data sources together, to get at that multi-model that we've all been talking about. I do think we're lacking there.

Adam Thomas:
There are — I've noticed there's been, with the exception of maybe one or two speakers, relatively little discussion of genomics. And given that the model that Dr. Gordon has suggested was to follow oncology, I'm wondering if people think that should be a priority.

Greg Simon:
This is Greg. Can't help but make a snarky comment, which is, we have to have some — we have to have some phenotypes that are all useful before we can go looking for genotypes. And I worry that, in our mental health space, we're still more, like, looking for a gene for shirt color than for eye color.

Roy Perlis:
It's — so, it's Roy here. So, first, I can't help but point out that the first depression genes actually came from using really crappy big-data phenotypes, in part from health records and in part from 23andMe. So, as much as we decry our psychiatric phenotypes, that did lead us to find the first genes. I also can't resist point out that the FDA lists more than 100 drugs on its own site, including probably 30-plus psychotropics, where genes are in the label.

I think — I completely agree with Greg that, in most — the — you know, I like to have my hand in how about polygene where it's due, but Greg is right that there is precious little to go on with genomics, with the [inaudible] very un-sexy P450 stuff that we've had for many years. So, as a field, we have P450, we know it affects drug levels, we know at the extremes drug levels affect response. So, I actually think, you know, in general, it's a combination of — no one wants to talk about P450 because it's old news, and no one wants to acknowledge that polygenic disease or phenotypes just haven't really worked very well in treatment response so far. So — and I'm agreeing with Greg, but so far, you know, the one counterexample is the one that no one really wants to do anything with because it's not sexy.

Male Speaker:
Right. That's useful.

Dylan Nielson:
All right, can I also — this is Dylan Nielson. Can I just jump in and comment that, if the — if the focus is purely on treatment response, as somebody said, the phenotypes get much, much clearer. If you're only concerned with whether or not the patient subjectively feels better after they've taken a medication or had a treatment, it's a clearer question in some ways than what particular diagnosis they may or may not have.

Male Speaker:

Sarah Lisanby:
Dylan, I want to thank you for making that comment and also encourage our other NIH and NIH colleagues who are on the line to feel free to participate in this discussion, because we've really brought together a lot of experts from different areas within NIH, as well. And so, your comment about — it brings us to the focus of our discussion here, which is how we can use data to inform treatment selection. And that focus is very specific.

Now, if getting a better diagnosis informs treatment selection, then that's part of it, but our main focus, where the rubber hits the road, what we — what — wanted to understand better is helping clinicians to select the right treatment for the right patient at the right time and really realize this vision of personalized medicine for psychiatry and for mental health. And so, as Dylan points out, if you have measurement showing before and after the person improved, that is a pretty clear phenotype, although I will say that you've got to decide what measure to use and reliably measure it and make that available. But — any comments on this focus on treatment, as opposed to diagnosis?

Peter Zandi:
So, Holly, one quick counterpoint to, the treatment outcome is easier to define. In some sense, it is; in some sense, it's not so easy. You were just alluding to it. Over one timeframe, somebody could have the — sort of a quick response, but are you looking over one year or two years? Some people have partial responses. And so, it's not so easy, actually, in these pharmacogenetic studies and these treatment prediction studies, to define the outcome in a clear and clean way. There are challenges to it. I don’t — I do think it's easier than maybe diagnosis, but it still comes with challenges about how to define a good outcome.

Sarah Lisanby:
Thank you for raising that, and I know we have developmental experts on this meeting. And when we think about defining outcomes, it is when and what stage of life because there may be interventions done early that have different impacts later on in development. And so, I want to remind ourselves not just to be thinking about treating diagnoses or clinical conditions once they occur, but should we be thinking about interventions in a pre-symptomatic period that might be preventative? Any thoughts about that?

Neal Ryan:
This is Neal. Obviously, you raise an issue dear to my heart, except usually the interval you're talking about is beyond the five-year interval for a study. So, they're a little bit hard to plan, [unintelligible], I think

Sarah Lisanby:
Well, just to remind us, in Dr. Gordon's introductory remarks, he did paint a wide vista, in terms of what timeframe one might envision doing projects like this. And the timeframe, as I recall, he painted was — could be beyond five years; could be, you know, 10 or 20 years; could be longer. We want to hear from you. What would it take? What — where should we be focusing? What are your thoughts about what would be the most impactful way to use data to inform care for mental health, and to transform mental healthcare? I encourage you to think really broad. Don't necessarily put yourself in a five-year box. Free yourself up to think broadly.

Neal Ryan:
That was the best being corrected I've ever experienced in the past long time, so this is good.

Sarah Lisanby:

David McMullen:
Holly, maybe while people think about that, could I ask a question?

Sarah Lisanby:
Go ahead.

David McMullen:
So, I think I heard "FDA" for maybe the first time a few minutes ago, in a little bit of a different way, and I apologize if someone else has talked about it during this workshop, but what do people think the long-term strategy is, and what regulation and oversight that the think the FDA should or should not be performing in terms of those clinical decision support systems?

Sarah Lisanby:
Yeah, I don't think that we actually have addressed that question yet, and so, to be — my — since we have a multidisciplinary audience here — it's David McMullen on the line — can you tell us more about what you mean by clinical decision support software and the role that the FDA devises branch has with respect to regulating that type of software?

David McMullen:
Sure. So, right now, this is something that the FDA is currently figuring out, but they do have draft guidance out on what they term clinical decision support. "Software as a medical device" might be another term for people here. But basically, if you have a product because they regulate medical claims, where you are stating that you can help diagnose [unintelligible] provide recommendations, then they are saying that they might have the regulatory oversight authority to regulate those. So, you have to come in, demonstrate safety and efficacy, and bear sort of an objective third-party body for that.

And so, I know I've heard different systems being developed, and I don't — I haven't heard people talking about selling any of them so far, but for the long-term rollout of this and for clinical translation. There will likely be an end product. And so, do people think about how to interact with the FDA or how to design the studies that they can be evaluated by them?

Roy Perlis:
So — Roy here. I think this I actually a place that NIMH can be extremely helpful as sort of an honest broker. Most investigators — most non-industry investigators don't have the resources or wherewithal to engage with FDA. Pharmacists have, you know, offices full of people to jump through the regulatory hurdles. But I think FDA's been clear with its software as a medical device guidance about what they want to do.

So, I think if NIMH — if we — NIMH wants to see more of these kinds of things that, you know, our one-funded investigators develop [inaudible] FDA, I think [inaudible] some support — some sort of mechanism to help investigators engage with FDA would be really helpful. As most of us are happy to get our models into the world, I don't have the time to hire FDA lawyers, and frankly, I'd rather put my software out in the world and let someone else worry about commercializing it, which is what's happened. Yeah, this is [inaudible], and I really could [inaudible].

Sarah Lisanby:
Great, thank you. So, the — we're — that's a concrete thing that could be put in place to help realize this vision. Just to underscore your saying, mechanisms to help investigators engage with the FDA around issues like software as a medical device, and what that regulatory path would be, as well as facilitating partnerships with other industries that would then take and support that software, which would not be supported solely in an academic lab. So, bridging with regulatory, bridging with companies, and facilitation of that bridging. Did I grab your point right?

Roy Perlis:
Yes, exactly, and, given that there's already been toes dipped in the water with other devices, I think software as a medical device is a good place to go next.

Greg Simon:
This is Greg. I was going to add, I'm fond of analogies, and one analogy I would like to use to talk about sort of the different stakeholders or players in this space is the weather world, because I'm a big weather nerd. And interestingly, in the weather world, all weather data in the world are pretty much in the public domain. We could go find them right now if we wanted, if we knew what to do with them. Then, there's a middle layer of modelers and analysts who develop weather models, which is an interesting mixed space. It includes, you know, nonprofits, academics.

It does include some for-profit people, but that layer — that layer is on top of weather data, and it — anyone can access all weather data, and a lot of people's models are out there in public for anyone to see. And then, there are meteorologists who communicate. You know, who work on television or on the radio or something like that. They are not modelers; they are not data nerds; by and large, they are communication experts.

At least, you know, the way I see it — in terms of the way, you know, my group and my team operate, see it — is, the data layer is the health systems. We are in that weather modeler layer. You know, we're trying to do things and put them in the public domain. There need to be some meteorologists in this space who are about implementation communication, and that is maybe a different skill set in a different business model. And those may be the people who bring those things to the FDA, and I am all for them.

So, I — you know, to me, it's an important sort of strategic question to think about, you know, which parts of this work line more in that for-profit world which is — you know, isn't — it works. You know, but I do need to acknowledge — and maybe I'm in the same boat as Roy to say, I'm going to live more in the atmospheric sciences world than in the meteorology sort of public communication world. And I would hope that some of those TV meteorologists would take the models that I built and put them to good use to inform the public about whether to have their picnic or not.

Sarah Lisanby:
Great. Well, that's a really nice metaphor. I want to do a time check. We're about at the end. We have about two minutes left, and I'd like to again ask if there's someone else that has not had a chance to make a comment. And they — if you've been holding back, this would be your invitation to do so. And as you — as you think about what comment — what that comment might be, I want to go ahead and thank our moderators; I want to thank all of our panelists. You've really covered a lot of ground is a short period of time, very helpful to us. I want to thank our listening audience from the public who have signed on to join us, and to let you know that this is the beginning.

We will be archiving this, we will be transcribing this, and we'll be learning from the information that you brought forward to help inform what needs to be put in place to be able to build towards a future where multiple data can inform treatment selection and transform mental healthcare. I want to go back to the words of Ken Duckworth, who had to step out. Obviously, a lot of pressure's underway in terms of the mental health needs of our — of our — the patients that we serve, now more than ever.

And we want to really provide a — not a fragmented, but an integrated and cohesive system of care that can help patients thrive and get the cure they need wherever and when they need it. So, let me just open it up to any last comments from my fellow program community members or anyone else. And, hearing none, I want to thank you all again. And we appreciate your input, and we are going to digest and think hard on what you've taught us today. I know I've learned a lot. And thank you for your — hanging with us until they end, and I'm going to go ahead and adjourn. Thank you so much. Be safe, stay well, and take care. Thank you.

Original Article

Blog Post » Piecing Together the Genetic Puzzle of Autism

One of the privileges I have had as director of the National Institute of Mental Health is chairing the Interagency Autism Coordinating Committee (IACC). The IACC, charged with coordinating activities across multiple federal agencies with regards to research, services, and policy around autism spectrum disorder (ASD), comprises federal and private partners. Particularly valuable are the contributions from members of the committee who have lived experience with ASD. We also hear regularly from members of the public, most of whom have ASD or care for someone with ASD. They all speak eloquently about the challenges associated with this developmental disorder of social cognition and behavior. Their experiences inspire our research portfolio aimed at supporting and enhancing the lives of individuals with ASD.

Fundamental to this effort are long-term investments in understanding the origins of ASD in early development, and central to developing such an understanding is deciphering the disorder’s genetic basis. Last month, I wrote about schizophrenia, a mental disorder that has a significant genetic component. I wrote about large-scale, NIMH-supported research efforts that are using techniques such as gene mapping and whole-exome sequencing to better understand the genetic component of schizophrenia. And I wrote about how these studies have demonstrated that it’s likely a combination of many small-effect-size genetic variants and a handful of large-effect-size genetic variants that increase a person’s chances of developing schizophrenia. These genetic findings offer crucial clues that can help unravel the neurobiology of the disorder and lead to novel treatments.

The story of the genetics of ASD is just as compelling and rapidly evolving. For some time, studies have suggested that large-effect-size genetic variants may play an influential role in autism. A new study, published this February and highlighted in the NIH Director’s Blog and popular press articles, extends these early findings. Using whole-exome sequencing, the research team — which included the NIH-funded Autism Sequencing Consortium — examined data from nearly 12,000 individuals with ASD and more than 23,000 individuals without ASD. The findings revealed 102 genes linked to the disorder, the result of rare mutations that aggregate in these genes. As I discussed in my previous message on schizophrenia genetics, identifying these genetic variants means that the pathway to investigating biological mechanisms and potential clinical targets is (relatively) straightforward.

Having made these genetic discoveries, the researchers proceeded to explore some fundamental questions about ASD. One major question in the field has been whether the neurobiological mechanisms underlying ASD differ depending on whether an individual also has an intellectual disability or other developmental delay. In this study, the researchers found that although many of the 102 genes are implicated in both ASD and other neurodevelopmental disorders, approximately half of the genes are altered more frequently in individuals with ASD who do not have a co-occurring developmental delay. These findings suggest that there may be a set of biological mechanisms underlying the characteristic social and behavioral features of ASD that are separate from, and in addition to, a set of mechanisms that underlies more general brain development.

Another fundamental question is why ASD is more frequently diagnosed in boys than in girls. One hypothesis is that unknown biological mechanisms may make girls more resilient to autism risk factors. In this study, the researchers found that the genetic variants they had identified were more than twice as frequent among girls with ASD than among boys with ASD. These data fit with the hypothesis that girls are more likely to have certain biological resilience factors and, as a result, it may take a greater burden of genetic risk for girls to receive an ASD diagnosis.

To learn more about the biology of ASD, the researchers also took a close look at what the ASD-associated genes do in the brain, and when and where in the brain they do it. Prior studies had suggested that genes linked with autism may play a role in the early development of the cortex, the outer layer of the brain that is critical to many important functions, including attention, thought, memory, and language. Having identified a much broader set of ASD-associated genes, the authors of this new study were able to make some interesting new observations. First, they confirmed that the vast majority of the 102 genes are expressed in the cortex, early in development. But they also found differences depending on whether the genes were primarily associated with ASD or with broader neurodevelopmental delay: genes associated with ASD were likely to be expressed in the developing brain soon after birth, while those associated with neurodevelopmental delay tended to be expressed even earlier, before birth.

The researchers also found that most of the 102 ASD-associated genes could be categorized as belonging to one of two broad mechanistic families: genes that play a role in turning other genes on and off and genes that are involved in synapses and other forms of neuronal communication. And finally, they found that these genes tended to be expressed in neurons belonging to both major classes of neuron types: excitatory and inhibitory neurons.

These exciting findings begin to write the story of the complex causal web that leads from genes to autism. The story seems to begin early in brain development. It involves the regulation of development through gene expression and the establishment of connections between neurons. And it culminates in different observable features (which we categorize as ASD or other neurodevelopmental delays) depending upon the genes involved. This is genuine progress, though there is much work left to be done. For starters, the authors point out that the type of rare variation they examined in these 102 genes explains only a small fraction of the variation in outcomes in the population.

We need to continue to work on understanding the myriad factors, genetic and environmental, that play a causal role in autism. And most importantly, we need to take this growing knowledge base and translate it into better treatments to improve the lives of individuals and families living with ASD.


Satterstrom, F. K., Kosmicki, J. A., Wang, J., Breen, M. S., De Rubeis, S., An, J.-Y., Peng, M., Collins, R., Grove, J., Klei, L., Stevens, C., Reichert, J., Mulhern, M. S., Artomov, M., Gerges, S., Sheppard, B., Xu, X., Bhaduri, A., Norman, U.,…Buxbaum, J. D. (2020). Large-scale exome sequencing study implicates both developmental and functional changes in the neurobiology of autism. Cell, 180(3), 568-584.e23. doi:10.1016/j.cell.2019.12.036

Original Article

Scientific Meeting » Webinar: Responding to the Alarm: Addressing Black Youth Suicide

Webinar: Responding to the Alarm: Addressing Black Youth Suicide


On April 21, hear from experts on black youth suicide in America during an interactive roundtable webinar hosted by the Office for Disparities Research and Workforce Diversity at the National Institute of Mental Health, in collaboration with the Office of Behavioral Health Equity at the Substance Abuse and Mental Health Services Administration.

The webinar will feature the co-authors of Ring the Alarm: The Crisis of Black Youth Suicide in America, a report from the Congressional Black Caucus’s Emergency Taskforce on Black Youth Suicide and Mental Health.

Panelists will discuss topics such as:

  • The latest trends and prevalence of suicide and suicide-related behaviors among black children and youth;
  • Common risk and protective factors that are most salient for this population;
  • Perspectives on this crisis and ways to identify black children and youth who are at risk for suicide; and
  • Thoughts on how to best engage this population and provide them with developmentally and culturally appropriate support and care.


Rhonda C. Boyd, Ph.D.
Associate Professor, Psychiatry, Perelman School of Medicine at the University of Pennsylvania

Michael Lindsey, Ph.D., MSW, MPH, Executive Director, McSilver Institute for Poverty Policy and Research, Constance and Martin Silver Professor of Poverty Studies

Arielle Sheftall, Ph.D.
Assistant Professor, The Ohio State University College of Medicine

Jason Wilson
Founder and CEO of The Yunion, Inc.


Please register online to attend this free webinar.

Original Article

Science News » Infant Temperament Predicts Personality More Than 20 Years Later

Researchers investigating how temperament shapes adult life-course outcomes have found that behavioral inhibition in infancy predicts a reserved, introverted personality at age 26. For those individuals who show sensitivity to making errors in adolescence, the findings indicated a higher risk for internalizing disorders (such as anxiety and depression) in adulthood. The study, funded by the National Institutes of Health and published in Proceedings of the National Academy of Sciences, provides robust evidence of the impact of infant temperament on adult outcomes.

“While many studies link early childhood behavior to risk for psychopathology, the findings in our study are unique,” said Daniel Pine, M.D., a study author and chief of the NIMH Section on Development and Affective Neuroscience. “This is because our study assessed temperament very early in life, linking it with outcomes occurring more than 20 years later through individual differences in neural processes.”

Temperament refers to biologically based individual differences in the way people emotionally and behaviorally respond to the world. During infancy, temperament serves as the foundation of later personality. One specific type of temperament, called behavioral inhibition (BI), is characterized by cautious, fearful, and avoidant behavior toward unfamiliar people, objects, and situations. BI has been found to be relatively stable across toddlerhood and childhood, and children with BI have been found to be at greater risk for developing social withdrawal and anxiety disorders than children without BI.

Although these findings hint at the long-term outcomes of inhibited childhood temperament, only two studies to date have followed inhibited children from early childhood to adulthood. The current study, conducted by researchers at the University of Maryland, College Park, the Catholic University of America, Washington, D.C., and the National Institute of Mental Health, recruited their participant sample at 4 months of age and characterized them for BI at 14 months (almost two years earlier than the previously published longitudinal studies). In addition, unlike the two previously published studies, the researchers included a neurophysiological measure to try to identify individual differences in risk for later psychopathology.

The researchers assessed the infants for BI at 14 months of age. At age 15, these participants returned to the lab to provide neurophysiological data. These neurophysiological measures were used to assess error-related negativity (ERN), which is a negative dip in the electrical signal recorded from the brain that occurs following incorrect responses on computerized tasks. Error-related negativity reflects the degree to which people are sensitive to errors. A larger error-related negativity signal has been associated with internalizing conditions such as anxiety, and a smaller error-related negativity has been associated with externalizing conditions such as impulsivity and substance use. The participants returned at age 26 for assessments of psychopathology, personality, social functioning, and education and employment outcomes.

“It is amazing that we have been able to keep in touch with this group of people over so many years. First their parents, and now they, continue to be interested and involved in the work,” said study author Nathan Fox, Ph.D., of the University of Maryland Department of Human Development and Quantitative Methodology.

The researchers found that BI at 14 months of age predicted, at age 26, a more reserved personality, fewer romantic relationships in the past 10 years, and lower social functioning with friends and family. BI at 14 months also predicted higher levels of internalizing psychopathology in adulthood, but only for those who also displayed larger error-related negativity signals at age 15. BI was not associated with externalizing general psychopathology or with education and employment outcomes.

This study highlights the enduring nature of early temperament on adult outcomes and suggests that neurophysiological markers such as error-related negativity may help identify individuals most at risk for developing internalizing psychopathology in adulthood.

“We have studied the biology of behavioral inhibition over time and it is clear that it has a profound effect influencing developmental outcome,” concluded Dr. Fox.

Although this study replicates and extends past research in this area, future work with larger and more diverse samples are needed to understand the generalizability of these findings.


Tang, A., Crawford, H., Morales, S., Degnan, K. A., Pine, D. S., & Fox, N. A. (2020). Infant behavioral inhibition predicts personality and social outcomes three decades later. Proceedings of the National Academy of Sciences.


MH093349; MH002782

About the National Institute of Mental Health (NIMH): The mission of the NIMH is to transform the understanding and treatment of mental illnesses through basic and clinical research, paving the way for prevention, recovery and cure. For more information, visit the NIMH website.

About the National Institutes of Health (NIH): NIH, the nation's medical research agency, includes 27 Institutes and Centers and is a component of the U.S. Department of Health and Human Services. NIH is the primary federal agency conducting and supporting basic, clinical, and translational medical research, and is investigating the causes, treatments, and cures for both common and rare diseases. For more information about NIH and its programs, visit the NIH website.

NIH…Turning Discovery Into Health®

Original Article

Blog Post » Coping with Coronavirus: Support for the Autism Community

April is Autism Awareness Month, although this April is quite different than the last. Many people’s lives — including those of individuals with autism and their families — have been dramatically altered by the coronavirus pandemic. Before coronavirus, I had written a Director’s Message focused on recent advances in autism genetics research. That will be coming later in the month. But first, I felt the need to address the challenges faced by individuals with autism and their families during this difficult time.

The coronavirus — and the subsequent response to slow the spread — has altered our lives, disrupting the routines, schedules, situations, and environments we’ve carefully crafted to help us learn, work, play, and grow. Individuals with autism, children and adults alike, often rely even more heavily on these structures. With social distancing requirements and shelter-in-place orders closing workplaces, houses of worship, schools, community centers, and more, many of the services people with autism rely on are no longer available. Changes in routine might lead some people with autism to experience increased anxiety or display disruptive or aggressive behavior; others may have trouble coping with the distress. Families and loved ones may also find it challenging to manage the additional stress brought about by changes in family routines that result from the coronavirus pandemic.

Like everyone, people with autism may experience anxiety caused by the uncertainty of our current situation and the disruption in comforting routines, sadness at the loss of social connectedness, and grief with the passing of friends and loved ones. But because autistic people experience social situations and changes in routine differently from many of us, we need to be aware that they may need extra attention and individualized support during this time.

Understanding how individuals with autism are coping with the pandemic is important. Fortunately, we’re already beginning to get some data. The Simons Foundation just published results about COVID-related impacts from a survey sent to participants in their SPARK (Simons Foundation Powering Autism Research for Knowledge) study. The SPARK study aims to collect medical and behavioral data from a group of 50,000 individuals with autism and their family members in the U.S. The results, based on the responses of 8,000 participants, are telling. Most respondents were parents of children with autism. The disruptions these families are experiencing are significant: Nearly all (98%) respondents reported that their children’s schools were closed, and 63% are missing key therapies. Only 42% said that their children with autism could understand information about COVID-19 “moderately” or better, possibly making it difficult for many children to understand why their daily activities are changing. Although most of the respondents said that their child was doing well overall, more than 90% reported worsened behavioral, mood, and/or anxiety symptoms. A second SPARK study survey, targeted at adults with autism, is currently underway.

There are resources to help. In particular, many federal agencies and organizations that support services for individuals and families affected by autism have rapidly set up webpages with information, ideas, and links to resources. The NIMH’s Office of Autism Research Coordination, which supports the Interagency Autism Coordinating Committee (IACC), has launched a Coronavirus Resources webpage that links to many of these resources, including sites with practical information on the virus itself, mental health resources, and resources people can use to address other specialized issues facing the autism community. The autism community has also been active in developing supportive resources. For example, Autism Speaks and the Autism Society both have organized online resources with practical toolkits for individuals with autism and their families to help them cope. Other groups, such as the Autism Science Foundation, have hosted podcasts to disseminate helpful information and online meetings for members of the autism community to discuss their concerns.

But what really works? Turning again to the SPARK survey, we have at least some ideas for parents who are caring for children or adults with autism. Taking breaks for rest and relaxation, telling stories, participating in hobbies, cooking for the family, spending time with family members both in person and virtually, and using technologies to connect with therapists and other caregivers were all mentioned as helpful ideas by survey respondents. These practices can also be helpful for independent autistic adults. In addition, autistic adults can access resources for assistance with coping with stress and anxiety, job loss or disruption, disruption in benefits, and management of physical or mental health conditions.

Finally, we have to remember that we are all in this fight against coronavirus, together. Therefore, this message is not only meant for those who have autism and their families and loved ones. All of us should recognize the additional challenges faced by individuals with autism and other disabilities, as well as their families and care providers. We should also be ready to reach out to those who may need extra support during this time. Remind them that they are not alone and that there are lots of ways they can get help if they need it.

Original Article